Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 6 out of 60 is failed (threshold=3) [....X......XX.X...........X.......X.........................] duration: 2.479866957s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:51:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.479985124s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 6 out of 60 is failed (threshold=3) [.....X......XX.X...........X.......X........................] duration: 2.577722721s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:51:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.577773311s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 6 out of 60 is failed (threshold=3) [......X......XX.X...........X.......X.......................] duration: 2.126215234s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:51:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.126247709s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.08758144s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:50:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.087609358s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [X......X......XX.X...........X.......X......................] duration: 2.188486117s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:50:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.188524258s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.288744929s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:50:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.288777671s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.13758488s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:50:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.137614481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [.X......X......XX.X...........X.......X.....................] duration: 2.526887546s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:50:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.526925803s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.086456594s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:49:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.086484609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [..X......X......XX.X...........X.......X....................] duration: 2.14877595s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:49:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.148796497s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.214929081s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:49:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.214971875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [X......X....................................................] duration: 1.132102193s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:49:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.132137194s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [...X......X......XX.X...........X.......X...................] duration: 2.576580947s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:49:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.576608818s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.267895532s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:49:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.267938476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [....X......X......XX.X...........X.......X..................] duration: 2.163137277s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:48:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.163159787s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X......X...................................................] duration: 1.064196714s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:48:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.06422989s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [.....X......X......XX.X...........X.......X.................] duration: 2.518214712s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:48:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.518247468s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 907.329713ms trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:47:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 907.372979ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [......X......X......XX.X...........X.......X................] duration: 2.181521834s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:47:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.181624104s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X......X..................................................] duration: 1.12838864s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:47:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.128420497s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.362278174s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:46:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.362310991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [.......X......X......XX.X...........X.......X...............] duration: 2.094199264s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:46:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.094223467s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X......X.................................................] duration: 701.906463ms trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:46:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 701.931245ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [........X......X......XX.X...........X.......X..............] duration: 2.566941169s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:45:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.567062295s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X......X................................................] duration: 1.279448277s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:45:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.279479426s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 928.804715ms trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:45:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 928.844189ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X......X...............................................] duration: 1.038681813s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:45:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.038731949s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [.........X......X......XX.X...........X.......X.............] duration: 2.610241435s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:44:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.610299004s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.012816904s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:44:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.012845623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X......X..............................................] duration: 500.352229ms trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:44:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 500.38046ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X......X.............................................] duration: 1.190851159s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:43:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.19090244s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.085792273s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:43:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.085830886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [..........X......X......XX.X...........X.......X............] duration: 2.140080131s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:43:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.140112867s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X......X............................................] duration: 918.597122ms trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:43:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 918.673099ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.29861139s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:43:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.298637772s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X......X...........................................] duration: 1.132950853s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:43:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.133070536s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [...........X......X......XX.X...........X.......X...........] duration: 2.133356666s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:42:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.133382494s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X......X..........................................] duration: 1.395022612s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:42:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.395049904s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 698.808139ms trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:41:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 698.838515ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [............X......X......XX.X...........X.......X..........] duration: 2.090123778s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:41:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.090153863s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [.............X......X......XX.X...........X.......X.........] duration: 2.138500274s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:41:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.138525756s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X......X.........................................] duration: 1.306841631s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:40:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.306873788s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [..............X......X......XX.X...........X.......X........] duration: 2.211669169s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:40:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.211700943s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..........X................................................] duration: 1.227877842s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:40:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.227910668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X......X........................................] duration: 1.159682116s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:40:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.159717556s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [...............X......X......XX.X...........X.......X.......] duration: 2.519508382s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:40:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.519552993s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X......X.......................................] duration: 732.923484ms trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:40:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 733.029729ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..........X...............................................] duration: 1.244632333s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:40:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.244665329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [................X......X......XX.X...........X.......X......] duration: 2.146822418s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:39:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.146852271s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..........X..............................................] duration: 1.262686804s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:39:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.262721493s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)