Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 401.688518ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:53:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 401.702297ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 422.071677ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:52:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 422.094325ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 380.531664ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:52:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 380.556143ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 433.532961ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:51:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 433.56047ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 398.292397ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:50:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 398.316904ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 363.506552ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:49:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 363.535436ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 407.288536ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:48:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.302273ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 544.395311ms trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:48:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 544.419833ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.024932614s trace: https://jaeger.dev.storj.io/trace/1a3af9ed903cc0bf FAIL (2025-12-07 09:48:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 09:47:44.837516370 (42h19m45.665224762s) [upload=43986]: Starting upload Dec 7 09:47:44.837614125 (42h19m45.665322519s) [upload=43986]: Done starting upload Dec 7 09:47:44.837688951 (42h19m45.665397335s) [upload=43986]: Uploading segments... Dec 7 09:47:44.837914794 (42h19m45.665623176s) [upload=43986]: Got next segment. Inline: false Dec 7 09:47:44.837969653 (42h19m45.665678033s) [upload=43986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 09:47:44.837978794 (42h19m45.665687176s) [upload=43986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 09:47:44.863355427 (42h19m45.691063821s) [seg_pos={0 0}, upload=43986]: Begin upload segment... Dec 7 09:47:44.863421674 (42h19m45.691130069s) [seg_pos={0 0}, upload=43986]: Done begin upload segment. Dec 7 09:47:44.863438335 (42h19m45.691146732s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.863468892 (42h19m45.691177292s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864389665 (42h19m45.692098064s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.864684112 (42h19m45.692392509s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865167481 (42h19m45.692875882s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.865672335 (42h19m45.693380726s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866093139 (42h19m45.693801531s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.866452454 (42h19m45.694160855s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867298009 (42h19m45.6950064s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.867580182 (42h19m45.695288583s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.868734514 (42h19m45.696442907s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869274514 (42h19m45.696982905s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.869722876 (42h19m45.697431271s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.871230042 (42h19m45.698938442s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.872365415 (42h19m45.700073817s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.874697895 (42h19m45.702406293s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Uploading piece... Dec 7 09:47:44.908899783 (42h19m45.736608183s) [noise=true , address=77.90.31.19:4841 , piece_id=GVHVAKFOJSZ6F633VTDF5XNI6HGQ3QNFZ5FR7NPLDCARGYWFGPOA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.916250469 (42h19m45.743958866s) [noise=true , address=77.90.31.21:4877 , piece_id=6MMH3CIMB2VIIXCFPQMQ5BGFPVADSRR4JGBYVH7H4OFONPPFCYPA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 09:47:44.950929370 (42h19m45.778637762s) [upload=43986]: Next returned nil segment Dec 7 09:47:44.950941250 (42h19m45.778649642s) [upload=43986]: Waiting for error group managing segments... Dec 7 09:47:44.958722471 (42h19m45.786430864s) [noise=true , address=145.79.227.27:4690 , piece_id=ISO5SH6HXTD4KQ4MLXXVJTCBCM2FAEOL5EC6CWOWRDM4RMIPIIHA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959651942 (42h19m45.787360333s) [noise=true , address=77.90.31.10:4520 , piece_id=3DTPH2JA6WXUIL6NRMDXF5MHZJ5IJRWNWKYZBOH65MNI5VTCJM5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.959848246 (42h19m45.787556638s) [noise=true , address=145.79.227.42:4121 , piece_id=EVSUDJRYDSVD7CGIPVO3DAL6DIPBMTHJGOTQFLLUVCYCUKZGXP5A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960187456 (42h19m45.787895847s) [noise=true , address=145.79.227.23:4546 , piece_id=JPY7DKUU7CZSL6UHZTNL5FRPJFBABNZANTQVHX3JETHE764IZE7A, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960542131 (42h19m45.788250523s) [noise=true , address=145.79.227.38:4977 , piece_id=E6ZYHRTQFQVT6MUYQP6S5N6IHKHP5CDKK3D5UAH7P5GTGOIMBZ3Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960581031 (42h19m45.788289423s) [noise=true , address=145.79.227.32:4836 , piece_id=WZF22MAPN5CZAQIWIXAN4AKRKGNOB7GZCUYUK74FWNBPXORQK7YQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.960938402 (42h19m45.788646796s) [noise=true , address=77.90.31.11:4554 , piece_id=MMNE4YSCCYA3LJ4K7FOGS37VCC5LVWD627CW43BXDPCNUYQE5RMQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961532534 (42h19m45.789240933s) [noise=true , address=145.79.227.26:4653 , piece_id=FJ3A2F6EUZ336XBX5J2F6PGB4G3NVW5CE6LMHKMWXBECCZDWVGCA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.961723121 (42h19m45.789431513s) [noise=true , address=145.79.227.31:4799 , piece_id=WOYO3K2GWBH27OZ22EXHT7GBJ6TMUSOIGP4LTETWV7Q4T3JOZBEQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962344473 (42h19m45.790052852s) [noise=true , address=145.79.227.44:4193 , piece_id=DBMRM4PJPN7MXCQEEA2CV7F2HIMGUWM47CBHB3F2JRWXJMV2FZLQ, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962427491 (42h19m45.790135882s) [noise=true , address=145.79.227.29:4942 , piece_id=2MZJJHF64OM2WTAMJXD7MTF77AXSUWZPOCLPZ6ST6R2K64D5LS4Q, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:47:44.962896172 (42h19m45.790604562s) [noise=true , address=145.79.227.23:4547 , piece_id=BGQCKP4RPLECNOFKEGYQJCSBQNFQ5VIZIBLFQ7WNHLEKMRSU4DKA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: <nil> Dec 7 09:48:14.804335910 (42h20m15.63204431s) [noise=true , address=77.90.31.21:4879 , piece_id=RPM5HWWOE3SNGJS44LIHTRKPSRMDHIKPG24KHVZ4MNCTO42AWREA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804432649 (42h20m15.632141041s) [noise=true , address=77.90.31.12:4590 , piece_id=EXFVFIOAG4WL4VJECDMKOAUKS7UG6Q5RC4CY24Q42KHOZSGB35GA, seg_pos={0 0}, upload=43986]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 09:48:14.804490405 (42h20m15.632198788s) [seg_pos={0 0}, upload=43986]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 09:48:14.827327232 (42h20m15.655035623s) [upload=43986]: Done uploading segments... Dec 7 09:48:14.827363026 (42h20m15.655071417s) [upload=43986]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.024951396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 392.92798ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:49:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.949472ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 480.095347ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:48:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 480.124703ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 371.904995ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:47:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 371.931517ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 453.660712ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:46:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.686751ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 547.493404ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:45:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 547.529672ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 546.712215ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:45:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 546.823659ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 434.19826ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:44:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.243426ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 376.666054ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:43:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 376.709844ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 404.513843ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:42:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 404.624183ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 401.003908ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:41:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 401.033805ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 414.356299ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:40:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 414.380832ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 535.673021ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:40:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 535.697898ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 450.376749ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:40:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.402247ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 534.740601ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:39:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 534.765375ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 405.710071ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:38:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 405.797051ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 461.580221ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:37:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 461.619229ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 443.685997ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:36:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 443.73795ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 520.497981ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:36:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 520.547653ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 560.687744ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:35:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 560.714574ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 449.685721ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:34:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 449.715877ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 439.042518ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:34:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.069466ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 502.802294ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:33:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 502.828659ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 588.090899ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:32:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 588.108327ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 460.413122ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:31:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 460.442766ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 440.579265ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:31:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 440.60608ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 631.953089ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:30:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 631.98336ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 394.374843ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:29:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 394.40104ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 379.999397ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:29:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 380.02354ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 505.928195ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:28:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 505.953892ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 456.943942ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:26:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.960821ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 362.426029ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:25:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 362.456608ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 417.510858ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:25:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.53649ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 422.530745ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:24:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 422.597008ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 492.37787ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:23:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 492.468923ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 480.149047ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:23:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 480.168222ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 544.760692ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:22:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 544.784145ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 496.451599ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:22:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 496.477084ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 232.836622ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:21:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 232.94645ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 379.414247ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:21:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 379.461846ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 251.601594ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:20:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 251.623348ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 483.567866ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:20:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 483.598426ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)