Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 479.179681ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:22:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 479.206303ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 441.905931ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:21:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.932912ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 494.967663ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:21:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 494.991375ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 383.90119ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:20:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 383.931357ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 540.675103ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:20:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 540.774238ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 367.587362ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:19:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 367.611338ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 415.181159ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:18:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 415.209471ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 386.262183ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:16:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 386.294434ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 473.562394ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:16:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 473.595921ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 392.90114ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:16:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.927144ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 546.634295ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:15:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 546.663337ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 363.263736ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:14:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 363.293317ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 363.580743ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:13:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 363.610323ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 379.903003ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:13:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 379.935622ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 385.423739ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:12:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 385.46741ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 467.28071ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:11:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.311814ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 360.915897ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:10:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 360.940167ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 503.347621ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:10:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 503.37821ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 394.019263ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:09:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 394.043512ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 534.282281ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:09:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 534.314383ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 380.169671ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:08:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 380.22389ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 360.633741ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:08:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 360.663428ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 397.451412ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:07:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 397.476689ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 415.142945ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:07:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 415.170356ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 467.244972ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:05:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.274346ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 410.659699ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:04:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.686417ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 511.491857ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:03:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 511.521125ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 399.629811ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:03:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 399.65666ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 392.093082ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:02:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.120079ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 467.092008ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:01:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.117777ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 549.75837ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:01:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 549.780803ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 384.008248ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:00:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 384.039154ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 410.236225ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:59:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.265703ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 400.818219ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:59:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 400.867339ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 375.649225ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:58:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 375.748944ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 380.147233ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:57:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 380.170497ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 477.053913ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:56:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 477.078975ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 563.486127ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:55:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 563.545045ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 515.823013ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:54:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 515.866983ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 410.592326ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:54:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.621416ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 401.688518ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:53:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 401.702297ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 422.071677ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:52:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 422.094325ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 380.531664ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:52:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 380.556143ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 433.532961ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:51:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 433.56047ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 398.292397ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:50:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 398.316904ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 363.506552ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:49:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 363.535436ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 407.288536ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:48:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.302273ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 544.395311ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:48:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 544.419833ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.024932614s trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:48:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.024951396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 392.92798ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:49:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.949472ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)