Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 387.382974ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:24:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 387.406294ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 381.190386ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:23:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 381.214384ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 479.179681ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:22:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 479.206303ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 441.905931ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:21:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.932912ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 494.967663ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:21:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 494.991375ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 383.90119ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:20:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 383.931357ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 540.675103ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:20:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 540.774238ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 367.587362ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:19:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 367.611338ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 415.181159ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:18:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 415.209471ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 386.262183ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:16:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 386.294434ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 473.562394ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:16:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 473.595921ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 392.90114ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:16:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.927144ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 546.634295ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:15:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 546.663337ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 363.263736ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:14:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 363.293317ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 363.580743ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:13:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 363.610323ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 379.903003ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:13:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 379.935622ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 385.423739ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:12:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 385.46741ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 467.28071ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:11:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.311814ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 360.915897ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:10:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 360.940167ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 503.347621ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:10:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 503.37821ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 394.019263ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:09:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 394.043512ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 534.282281ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:09:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 534.314383ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 380.169671ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:08:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 380.22389ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 360.633741ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:08:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 360.663428ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 397.451412ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:07:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 397.476689ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 415.142945ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:07:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 415.170356ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 467.244972ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:05:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.274346ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 410.659699ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:04:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.686417ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 511.491857ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:03:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 511.521125ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 399.629811ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:03:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 399.65666ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 392.093082ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:02:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.120079ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 467.092008ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:01:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.117777ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 549.75837ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:01:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 549.780803ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 384.008248ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 10:00:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 384.039154ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 410.236225ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:59:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.265703ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 400.818219ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:59:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 400.867339ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 375.649225ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:58:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 375.748944ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 380.147233ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:57:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 380.170497ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 477.053913ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:56:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 477.078975ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 563.486127ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:55:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 563.545045ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 515.823013ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:54:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 515.866983ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 410.592326ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:54:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.621416ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 401.688518ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:53:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 401.702297ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 422.071677ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:52:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 422.094325ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 380.531664ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:52:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 380.556143ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 433.532961ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:51:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 433.56047ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 398.292397ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:50:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 398.316904ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 363.506552ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:49:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 363.535436ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 407.288536ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:48:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.302273ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 544.395311ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:48:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 544.419833ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)