Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X......X......................................] duration: 1.084423739s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:39:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.084469544s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [.................X......X......XX.X...........X.......X.....] duration: 2.602224597s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:39:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.602245664s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..........X.............................................] duration: 1.135781415s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:38:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.135815553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X......X.....................................] duration: 1.218022592s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:38:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.218118237s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [..................X......X......XX.X...........X.......X....] duration: 2.496895653s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:38:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.49692652s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..........X............................................] duration: 1.307690915s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:37:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.307721179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [...................X......X......XX.X...........X.......X...] duration: 2.116724122s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:37:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.116754331s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X....X..........X...........................................] duration: 1.225839725s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:37:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.225865633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X......X....................................] duration: 665.878689ms trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:37:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 665.912692ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [....................X......X......XX.X...........X.......X..] duration: 2.594342609s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:37:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.594370703s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X....X..........X..........................................] duration: 1.108846136s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:36:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.108874499s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X......X...................................] duration: 1.163195906s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:36:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.163223907s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X....X..........X.........................................] duration: 1.322309119s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:36:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.322335339s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [.....................X......X......XX.X...........X.......X.] duration: 2.304667173s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:36:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 2.304688072s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 7 out of 60 is failed (threshold=3) [......................X......X......XX.X...........X.......X] duration: 1m0.104911085s trace: https://jaeger.dev.storj.io/trace/78c3ad63aac6b527 FAIL (2026-03-10 19:36:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 23) ======LOGS====== :03.719970769 (104h35m1.316741801s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.720006314 (104h35m1.316777347s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.721245847 (104h35m1.318016883s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722131967 (104h35m1.318902999s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.722154312 (104h35m1.318925349s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.723422869 (104h35m1.320193903s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.724205203 (104h35m1.320976237s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:03.727803141 (104h35m1.324574176s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Uploading piece... Mar 10 19:35:04.167055423 (104h35m1.763826466s) [upload=117052]: Next returned nil segment Mar 10 19:35:04.167072399 (104h35m1.763843424s) [upload=117052]: Waiting for error group managing segments... Mar 10 19:35:04.350286979 (104h35m1.947058004s) [noise=true , address=81.171.14.70:4035 , piece_id=RNJYIOH4ETYQBSZFYTV3FX73J3PNFR2KBYNZEDXCVBOYMUAN5FCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.351883208 (104h35m1.948654233s) [noise=true , address=81.171.14.70:4043 , piece_id=LSOHRZMEV5TEQUKZIJPYHVP43CIZKMXBTRVKIZIQNSSFNMAA5FLQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.356149085 (104h35m1.952920098s) [noise=true , address=185.167.204.205:4036 , piece_id=BBGCJ7U7PQDNB5NZTJWHUWXR6TFYF6D7LOR2UUBNV647YQX5JD2Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.369094269 (104h35m1.965865391s) [noise=true , address=89.149.201.147:4011 , piece_id=YHPLXK3ADD6VMYDWQ55PLROBCY2IOLOR2ZWEZ22FRRA4TXKSSWIQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.382982597 (104h35m1.979753611s) [noise=true , address=81.171.14.70:4034 , piece_id=BPXMDI54RTQVH5QKY6JCN5H3AKCPH62V76VBES4GQ6SLIZDIMMOA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.384672279 (104h35m1.981443307s) [noise=true , address=81.171.14.70:4047 , piece_id=DLCUNYJFEKIU3MWFXK24QGJ4GW2H5XE4Y2ARNL2W55EZUMSC4Z7Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.386337288 (104h35m1.98310832s) [noise=true , address=81.171.14.70:4025 , piece_id=R62A3AXM4OKUZFZJ2LB55UPB6VRGHBQLMXXVBWDTHUDWPGPIWEVQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.387583761 (104h35m1.984354793s) [noise=true , address=89.149.201.147:4009 , piece_id=XFSD3W6JWUXGXTHYE57SS5E5322JTRG7RKBI7MBF25GU7SEAKZJQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.388110220 (104h35m1.984881248s) [noise=true , address=81.171.14.70:4026 , piece_id=4CBQ4RZ44USXNYAGA3CIGNC3LBQXR675JVJX6CVTV3X7KVYPRHQA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.389451350 (104h35m1.986222362s) [noise=true , address=185.167.204.205:4012 , piece_id=7UFJIUKCHPKYZF42HTP6Q5D2FMHRSCRUQ7MH2MYJSEOZHZAJRZXA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.404093624 (104h35m2.00086464s) [noise=true , address=81.171.14.70:4048 , piece_id=4F2F4X4ZJ4FXUYQJQKB7YB3JZY53QNVRUSM3N4HQKGEYHMFYQB2A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.413111656 (104h35m2.009882687s) [noise=true , address=185.167.204.206:4071 , piece_id=ZQJLN55G37453GZS54ZDTPM7CEU5GH6LJR3BIKSJDC3C4QOKW5YQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.416183369 (104h35m2.012954404s) [noise=true , address=89.149.201.147:4014 , piece_id=WZOPRPDLFVOO6O2JYZZB75CLIAA4SVM7OEZXBX6WOJV3PUL2YSAQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.449375785 (104h35m2.046146815s) [noise=true , address=185.167.204.206:4039 , piece_id=DKKIZZMA3RAKMLUJTU6OANGBA3F6XWRD5ELVNXAGBIOZMFBVQWZQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.457716093 (104h35m2.054487108s) [noise=true , address=185.167.204.205:4024 , piece_id=3CKPLBO33LOMNYBF47K2KGQFPO53ZXLMKRKWPVSR4FKQKXKMZMCA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.491116161 (104h35m2.087887192s) [noise=true , address=185.167.204.205:4005 , piece_id=E7RO77MCAMFEAIE7ETPALCBV5S4X3DG5ZUD27ZE55RXYICIIXNEQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.507099744 (104h35m2.103870782s) [noise=true , address=185.167.204.205:4029 , piece_id=R33ONZELVLYL76YTJKOZASDJKFBACNNMC3PDUIITQRDGBWWZUNKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.537139239 (104h35m2.133910269s) [noise=true , address=185.167.204.206:4067 , piece_id=RH2DWZ3FL7JX6EJ2WTJBPBSWDYVSHLYSLZCMMGRQXUDR6QXVZZ5A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.787100189 (104h35m2.383871224s) [noise=true , address=81.171.14.70:4033 , piece_id=ZFV477QXRX42OJIG4D5JLOYU3WERTD65PMHI5MFQLRTQJG3ENUTA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:04.962846444 (104h35m2.559617469s) [noise=true , address=89.149.201.147:4004 , piece_id=TMKNRIR6RWRW3TUSA2MQL4LCNTJ3LKEFIT6BAVRX2V54QHNF26FA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.299132133 (104h35m5.895903147s) [noise=true , address=185.167.204.205:4028 , piece_id=IQ4CMHEIUNHIUBQV2PVFQ6BURDMWU2S55VGFP5Y7HPFSDVBY6PYA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:08.481127955 (104h35m6.077898968s) [noise=true , address=185.167.204.206:4051 , piece_id=GO4OTIL5WFOS7V77FAT2NRFG2HRE2HXKMU2C3XEGDBGPQL5U5HUA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:35:10.314952727 (104h35m7.91172374s) [noise=true , address=185.167.204.206:4046 , piece_id=5WVC4R6VY25QUMGSOAORUK3WPGIADBXLKN7ZSLJ4GQEVKMSHZPKA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: <nil> Mar 10 19:36:03.322697363 (104h36m0.919468398s) [noise=true , address=185.167.204.206:4062 , piece_id=7NP2ZIMW6ZDXW3YHZSBO5PRTAZPLYLDWXDA4QPN5XH2C6KAGYLEA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12BL6XLqsoUGm2FxrTgYTxLZakt2u2CAhZJG82oJ31eaE4vTtMr, address:185.167.204.206:4062): protocol: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41266->185.167.204.206:4062: use of closed network connection Mar 10 19:36:03.322771647 (104h36m0.919542676s) [noise=true , address=89.149.201.147:4007 , piece_id=AAJY5QBN55ENHTODH25WBVCFDT2SAQ2F27TKHFCIURCZSBMGOY6A, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:54460->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.322958851 (104h36m0.919729884s) [noise=true , address=185.167.204.205:4033 , piece_id=6JIW66HORXSEPSDWIUONOUZNA2UNXKBA4A6VZ7PBWW3IN2SCJSPQ, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:12Q73mXHLEfuX4VUbJ8PTjhMoXCrYSKjJbFxqjDdqF3U99N8FXe, address:185.167.204.205:4033): protocol: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47916->185.167.204.205:4033: use of closed network connection Mar 10 19:36:03.323134221 (104h36m0.919905244s) [noise=true , address=89.149.201.147:4018 , piece_id=V3KLJ75HJ3NRMBP2FIM33GC45QW4U5XMN2R4IAHSX32COGBJF47Q, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua, address:89.149.201.147:4018): protocol: read tcp 159.223.143.39:47122->89.149.201.147:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323461923 (104h36m0.920232956s) [noise=true , address=185.167.204.205:4022 , piece_id=F5UFUTAFPRBWMSU3QKV7ZSYQR6UUC7SBUDOC7AAYGDXTC5VYUBHA, seg_pos={0 0}, upload=117052]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: read tcp 159.223.143.39:51226->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:36:03.323503666 (104h36m0.920274689s) [seg_pos={0 0}, upload=117052]: Done waiting for segment. successful: 23 optimal: 26 errs: <nil> Mar 10 19:36:03.424787357 (104h36m1.021558384s) [upload=117052]: Done uploading segments... Mar 10 19:36:03.424860036 (104h36m1.021631063s) [upload=117052]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 23) Duration: 1m0.104938196s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X......X..................................] duration: 1.323149553s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:35:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.323180695s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X....X..........X........................................] duration: 1.230828438s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:35:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.230872252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X....X..........X.......................................] duration: 1.008745217s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:34:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.008775527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X......X.................................] duration: 1.134897377s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:34:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.134925051s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 6 out of 60 is failed (threshold=3) [.......................X......X......XX.X...........X.......] duration: 2.143259943s trace: https://jaeger.dev.storj.io/trace/3b4460442365c0e7 FAIL (2026-03-10 19:34:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389309232 (104h27m50.986080258s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389816473 (104h27m50.986587499s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.391931427 (104h27m50.988702464s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.394396629 (104h27m50.991167767s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.397263419 (104h27m50.994034452s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.399366031 (104h27m50.996137061s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.406080001 (104h27m51.002851053s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.409602698 (104h27m51.006373731s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.416502293 (104h27m51.013273329s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.420753887 (104h27m51.017524921s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.423262462 (104h27m51.020033496s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.823101372 (104h27m51.419872405s) [upload=116916]: Next returned nil segment Mar 10 19:27:53.823120446 (104h27m51.419891481s) [upload=116916]: Waiting for error group managing segments... Mar 10 19:27:54.000664652 (104h27m51.597435708s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.008012848 (104h27m51.604783892s) [noise=true , address=81.171.14.70:4029 , piece_id=4LYITJ27FYZX25WUL42RFJUX7VSDBA7CXJJASFKRAFGPXJYKM6QA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.011975049 (104h27m51.608746079s) [noise=true , address=81.171.14.70:4038 , piece_id=2HBQKDA5TGBX2P7NASC6NA4MWD7NVDUUG3VBAECSTG5TRDDR2I3Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013324482 (104h27m51.610095512s) [noise=true , address=185.167.204.205:4032 , piece_id=VXAHZM6L7462VFYIJFBL6RHMJAYOGY3EODRUOHWII4HKM255UGZA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013614745 (104h27m51.610385775s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.026623408 (104h27m51.623394435s) [noise=true , address=81.171.14.70:4033 , piece_id=LWM6DL6LLEPLGWK3Q3QIYLRYEJA4FWT7OUN5WPSRXPERNKFZKIIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.035560337 (104h27m51.63233135s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039223932 (104h27m51.635994945s) [noise=true , address=185.167.204.206:4052 , piece_id=AXXKU7XCFVOYI4KBLQ4P5JOCNVWTR27HDBB6LLCFMRFEKRE3VOQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039642416 (104h27m51.636413439s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.041029145 (104h27m51.637800171s) [noise=true , address=81.171.14.70:4047 , piece_id=DRMYHFSFKE6W754ZVGR4443E4IN3RSLIDF4YVS4JLRWTHJJSGNRA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.044680567 (104h27m51.64145159s) [noise=true , address=81.171.14.70:4037 , piece_id=U5Z5435XRWAWRIAS5HFPZI3YHGPRVWPRQB2BSOCLZDO3NGTMKO5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.050626247 (104h27m51.647397275s) [noise=true , address=185.167.204.206:4058 , piece_id=NEUNBF5MTH7ZNJ3OKD2SCMOSILZ7THBMO6EYNHZ4IQNIQ5T3IJ7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.063548076 (104h27m51.6603191s) [noise=true , address=185.167.204.205:4014 , piece_id=7JROEIOUUMNHTLLV3S4D3Q2RJ6JEZZBF6GHJI3BE635Z7NRRPPYA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.080832704 (104h27m51.677603718s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.088180011 (104h27m51.684951025s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.109755006 (104h27m51.706526034s) [noise=true , address=185.167.204.205:4035 , piece_id=PPE2DXQIF574R55F7BGW5SRV2MTAROPD2QERBVJ6IXKV4HMCIHFA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.117984644 (104h27m51.71475567s) [noise=true , address=185.167.204.205:4019 , piece_id=FXPKM5PNNZC5SEE3PBR7VPGX4Y644QCC6IBRJTF4PVRVUEC7XEQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.118587921 (104h27m51.715358933s) [noise=true , address=185.167.204.206:4040 , piece_id=WBPAL2X62XGHRZ6KPNSHAXNE4U5XDXYTESCTVBW6F4G4JAF5JALQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.127117193 (104h27m51.723888222s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.166283722 (104h27m51.763054748s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.472291315 (104h27m52.069062329s) [noise=true , address=81.171.14.70:4040 , piece_id=GIBXDZKK4H5WKV4AR2VAA6BWYNPN3AYOD66DN4WGWZPXKXHRRW5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.486254788 (104h27m52.083025819s) [noise=true , address=89.149.201.147:4013 , piece_id=X3IEGLOWJDQTXWRFFTEJGZFBD5TQJNURDIHP2RUGJE65YJUQAISQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.552368074 (104h27m52.149139108s) [noise=true , address=89.149.201.147:4023 , piece_id=JWOFVPWTRQ45HXXXKDRH2MDBKQL7MC6VNMBSLPBVJNDHDBE5XTAA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:56.435632635 (104h27m54.032403649s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:58.800969733 (104h27m56.397740758s) [noise=true , address=185.167.204.205:4033 , piece_id=O2SIBP4ZSONAV6MUS6IDXXNM7PDLJQBQBELUOUBXZOSKLDSIR7NA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:28:53.009605827 (104h28m50.606376886s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: read tcp 159.223.143.39:34090->89.149.201.147:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009693883 (104h28m50.606464943s) [noise=true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: read tcp 159.223.143.39:33010->89.149.201.147:4017: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009755582 (104h28m50.606526616s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 159.223.143.39:48940->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009829068 (104h28m50.606600111s) [seg_pos={0 0}, upload=116916]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:28:53.106470711 (104h28m50.703241735s) [upload=116916]: Done uploading segments... Mar 10 19:28:53.106515494 (104h28m50.703286524s) [upload=116916]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.143634352s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...................X......X................................] duration: 1.131696403s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:33:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.131725684s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X....X..........X......................................] duration: 1.25783623s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:33:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.257866174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X...................X......X...............................] duration: 1.214136714s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:33:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.214162248s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 6 out of 60 is failed (threshold=3) [........................X......X......XX.X...........X......] duration: 2.165086311s trace: https://jaeger.dev.storj.io/trace/3b4460442365c0e7 FAIL (2026-03-10 19:33:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389309232 (104h27m50.986080258s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389816473 (104h27m50.986587499s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.391931427 (104h27m50.988702464s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.394396629 (104h27m50.991167767s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.397263419 (104h27m50.994034452s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.399366031 (104h27m50.996137061s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.406080001 (104h27m51.002851053s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.409602698 (104h27m51.006373731s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.416502293 (104h27m51.013273329s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.420753887 (104h27m51.017524921s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.423262462 (104h27m51.020033496s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.823101372 (104h27m51.419872405s) [upload=116916]: Next returned nil segment Mar 10 19:27:53.823120446 (104h27m51.419891481s) [upload=116916]: Waiting for error group managing segments... Mar 10 19:27:54.000664652 (104h27m51.597435708s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.008012848 (104h27m51.604783892s) [noise=true , address=81.171.14.70:4029 , piece_id=4LYITJ27FYZX25WUL42RFJUX7VSDBA7CXJJASFKRAFGPXJYKM6QA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.011975049 (104h27m51.608746079s) [noise=true , address=81.171.14.70:4038 , piece_id=2HBQKDA5TGBX2P7NASC6NA4MWD7NVDUUG3VBAECSTG5TRDDR2I3Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013324482 (104h27m51.610095512s) [noise=true , address=185.167.204.205:4032 , piece_id=VXAHZM6L7462VFYIJFBL6RHMJAYOGY3EODRUOHWII4HKM255UGZA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013614745 (104h27m51.610385775s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.026623408 (104h27m51.623394435s) [noise=true , address=81.171.14.70:4033 , piece_id=LWM6DL6LLEPLGWK3Q3QIYLRYEJA4FWT7OUN5WPSRXPERNKFZKIIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.035560337 (104h27m51.63233135s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039223932 (104h27m51.635994945s) [noise=true , address=185.167.204.206:4052 , piece_id=AXXKU7XCFVOYI4KBLQ4P5JOCNVWTR27HDBB6LLCFMRFEKRE3VOQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039642416 (104h27m51.636413439s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.041029145 (104h27m51.637800171s) [noise=true , address=81.171.14.70:4047 , piece_id=DRMYHFSFKE6W754ZVGR4443E4IN3RSLIDF4YVS4JLRWTHJJSGNRA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.044680567 (104h27m51.64145159s) [noise=true , address=81.171.14.70:4037 , piece_id=U5Z5435XRWAWRIAS5HFPZI3YHGPRVWPRQB2BSOCLZDO3NGTMKO5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.050626247 (104h27m51.647397275s) [noise=true , address=185.167.204.206:4058 , piece_id=NEUNBF5MTH7ZNJ3OKD2SCMOSILZ7THBMO6EYNHZ4IQNIQ5T3IJ7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.063548076 (104h27m51.6603191s) [noise=true , address=185.167.204.205:4014 , piece_id=7JROEIOUUMNHTLLV3S4D3Q2RJ6JEZZBF6GHJI3BE635Z7NRRPPYA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.080832704 (104h27m51.677603718s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.088180011 (104h27m51.684951025s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.109755006 (104h27m51.706526034s) [noise=true , address=185.167.204.205:4035 , piece_id=PPE2DXQIF574R55F7BGW5SRV2MTAROPD2QERBVJ6IXKV4HMCIHFA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.117984644 (104h27m51.71475567s) [noise=true , address=185.167.204.205:4019 , piece_id=FXPKM5PNNZC5SEE3PBR7VPGX4Y644QCC6IBRJTF4PVRVUEC7XEQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.118587921 (104h27m51.715358933s) [noise=true , address=185.167.204.206:4040 , piece_id=WBPAL2X62XGHRZ6KPNSHAXNE4U5XDXYTESCTVBW6F4G4JAF5JALQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.127117193 (104h27m51.723888222s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.166283722 (104h27m51.763054748s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.472291315 (104h27m52.069062329s) [noise=true , address=81.171.14.70:4040 , piece_id=GIBXDZKK4H5WKV4AR2VAA6BWYNPN3AYOD66DN4WGWZPXKXHRRW5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.486254788 (104h27m52.083025819s) [noise=true , address=89.149.201.147:4013 , piece_id=X3IEGLOWJDQTXWRFFTEJGZFBD5TQJNURDIHP2RUGJE65YJUQAISQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.552368074 (104h27m52.149139108s) [noise=true , address=89.149.201.147:4023 , piece_id=JWOFVPWTRQ45HXXXKDRH2MDBKQL7MC6VNMBSLPBVJNDHDBE5XTAA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:56.435632635 (104h27m54.032403649s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:58.800969733 (104h27m56.397740758s) [noise=true , address=185.167.204.205:4033 , piece_id=O2SIBP4ZSONAV6MUS6IDXXNM7PDLJQBQBELUOUBXZOSKLDSIR7NA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:28:53.009605827 (104h28m50.606376886s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: read tcp 159.223.143.39:34090->89.149.201.147:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009693883 (104h28m50.606464943s) [noise=true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: read tcp 159.223.143.39:33010->89.149.201.147:4017: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009755582 (104h28m50.606526616s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 159.223.143.39:48940->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009829068 (104h28m50.606600111s) [seg_pos={0 0}, upload=116916]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:28:53.106470711 (104h28m50.703241735s) [upload=116916]: Done uploading segments... Mar 10 19:28:53.106515494 (104h28m50.703286524s) [upload=116916]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.165120911s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X....X..........X.....................................] duration: 1.25416647s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:33:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.254277384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X...................X......X..............................] duration: 1.220925881s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:32:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.220968393s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 6 out of 60 is failed (threshold=3) [.........................X......X......XX.X...........X.....] duration: 2.113233668s trace: https://jaeger.dev.storj.io/trace/3b4460442365c0e7 FAIL (2026-03-10 19:32:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389309232 (104h27m50.986080258s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389816473 (104h27m50.986587499s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.391931427 (104h27m50.988702464s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.394396629 (104h27m50.991167767s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.397263419 (104h27m50.994034452s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.399366031 (104h27m50.996137061s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.406080001 (104h27m51.002851053s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.409602698 (104h27m51.006373731s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.416502293 (104h27m51.013273329s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.420753887 (104h27m51.017524921s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.423262462 (104h27m51.020033496s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.823101372 (104h27m51.419872405s) [upload=116916]: Next returned nil segment Mar 10 19:27:53.823120446 (104h27m51.419891481s) [upload=116916]: Waiting for error group managing segments... Mar 10 19:27:54.000664652 (104h27m51.597435708s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.008012848 (104h27m51.604783892s) [noise=true , address=81.171.14.70:4029 , piece_id=4LYITJ27FYZX25WUL42RFJUX7VSDBA7CXJJASFKRAFGPXJYKM6QA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.011975049 (104h27m51.608746079s) [noise=true , address=81.171.14.70:4038 , piece_id=2HBQKDA5TGBX2P7NASC6NA4MWD7NVDUUG3VBAECSTG5TRDDR2I3Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013324482 (104h27m51.610095512s) [noise=true , address=185.167.204.205:4032 , piece_id=VXAHZM6L7462VFYIJFBL6RHMJAYOGY3EODRUOHWII4HKM255UGZA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013614745 (104h27m51.610385775s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.026623408 (104h27m51.623394435s) [noise=true , address=81.171.14.70:4033 , piece_id=LWM6DL6LLEPLGWK3Q3QIYLRYEJA4FWT7OUN5WPSRXPERNKFZKIIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.035560337 (104h27m51.63233135s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039223932 (104h27m51.635994945s) [noise=true , address=185.167.204.206:4052 , piece_id=AXXKU7XCFVOYI4KBLQ4P5JOCNVWTR27HDBB6LLCFMRFEKRE3VOQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039642416 (104h27m51.636413439s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.041029145 (104h27m51.637800171s) [noise=true , address=81.171.14.70:4047 , piece_id=DRMYHFSFKE6W754ZVGR4443E4IN3RSLIDF4YVS4JLRWTHJJSGNRA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.044680567 (104h27m51.64145159s) [noise=true , address=81.171.14.70:4037 , piece_id=U5Z5435XRWAWRIAS5HFPZI3YHGPRVWPRQB2BSOCLZDO3NGTMKO5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.050626247 (104h27m51.647397275s) [noise=true , address=185.167.204.206:4058 , piece_id=NEUNBF5MTH7ZNJ3OKD2SCMOSILZ7THBMO6EYNHZ4IQNIQ5T3IJ7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.063548076 (104h27m51.6603191s) [noise=true , address=185.167.204.205:4014 , piece_id=7JROEIOUUMNHTLLV3S4D3Q2RJ6JEZZBF6GHJI3BE635Z7NRRPPYA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.080832704 (104h27m51.677603718s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.088180011 (104h27m51.684951025s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.109755006 (104h27m51.706526034s) [noise=true , address=185.167.204.205:4035 , piece_id=PPE2DXQIF574R55F7BGW5SRV2MTAROPD2QERBVJ6IXKV4HMCIHFA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.117984644 (104h27m51.71475567s) [noise=true , address=185.167.204.205:4019 , piece_id=FXPKM5PNNZC5SEE3PBR7VPGX4Y644QCC6IBRJTF4PVRVUEC7XEQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.118587921 (104h27m51.715358933s) [noise=true , address=185.167.204.206:4040 , piece_id=WBPAL2X62XGHRZ6KPNSHAXNE4U5XDXYTESCTVBW6F4G4JAF5JALQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.127117193 (104h27m51.723888222s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.166283722 (104h27m51.763054748s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.472291315 (104h27m52.069062329s) [noise=true , address=81.171.14.70:4040 , piece_id=GIBXDZKK4H5WKV4AR2VAA6BWYNPN3AYOD66DN4WGWZPXKXHRRW5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.486254788 (104h27m52.083025819s) [noise=true , address=89.149.201.147:4013 , piece_id=X3IEGLOWJDQTXWRFFTEJGZFBD5TQJNURDIHP2RUGJE65YJUQAISQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.552368074 (104h27m52.149139108s) [noise=true , address=89.149.201.147:4023 , piece_id=JWOFVPWTRQ45HXXXKDRH2MDBKQL7MC6VNMBSLPBVJNDHDBE5XTAA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:56.435632635 (104h27m54.032403649s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:58.800969733 (104h27m56.397740758s) [noise=true , address=185.167.204.205:4033 , piece_id=O2SIBP4ZSONAV6MUS6IDXXNM7PDLJQBQBELUOUBXZOSKLDSIR7NA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:28:53.009605827 (104h28m50.606376886s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: read tcp 159.223.143.39:34090->89.149.201.147:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009693883 (104h28m50.606464943s) [noise=true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: read tcp 159.223.143.39:33010->89.149.201.147:4017: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009755582 (104h28m50.606526616s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 159.223.143.39:48940->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009829068 (104h28m50.606600111s) [seg_pos={0 0}, upload=116916]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:28:53.106470711 (104h28m50.703241735s) [upload=116916]: Done uploading segments... Mar 10 19:28:53.106515494 (104h28m50.703286524s) [upload=116916]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.113328116s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X...................X......X.............................] duration: 1.158132622s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:32:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.158157684s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X....X..........X....................................] duration: 1.155867196s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:32:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.155913336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 6 out of 60 is failed (threshold=3) [..........................X......X......XX.X...........X....] duration: 2.156263971s trace: https://jaeger.dev.storj.io/trace/3b4460442365c0e7 FAIL (2026-03-10 19:31:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389309232 (104h27m50.986080258s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389816473 (104h27m50.986587499s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.391931427 (104h27m50.988702464s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.394396629 (104h27m50.991167767s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.397263419 (104h27m50.994034452s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.399366031 (104h27m50.996137061s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.406080001 (104h27m51.002851053s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.409602698 (104h27m51.006373731s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.416502293 (104h27m51.013273329s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.420753887 (104h27m51.017524921s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.423262462 (104h27m51.020033496s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.823101372 (104h27m51.419872405s) [upload=116916]: Next returned nil segment Mar 10 19:27:53.823120446 (104h27m51.419891481s) [upload=116916]: Waiting for error group managing segments... Mar 10 19:27:54.000664652 (104h27m51.597435708s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.008012848 (104h27m51.604783892s) [noise=true , address=81.171.14.70:4029 , piece_id=4LYITJ27FYZX25WUL42RFJUX7VSDBA7CXJJASFKRAFGPXJYKM6QA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.011975049 (104h27m51.608746079s) [noise=true , address=81.171.14.70:4038 , piece_id=2HBQKDA5TGBX2P7NASC6NA4MWD7NVDUUG3VBAECSTG5TRDDR2I3Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013324482 (104h27m51.610095512s) [noise=true , address=185.167.204.205:4032 , piece_id=VXAHZM6L7462VFYIJFBL6RHMJAYOGY3EODRUOHWII4HKM255UGZA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013614745 (104h27m51.610385775s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.026623408 (104h27m51.623394435s) [noise=true , address=81.171.14.70:4033 , piece_id=LWM6DL6LLEPLGWK3Q3QIYLRYEJA4FWT7OUN5WPSRXPERNKFZKIIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.035560337 (104h27m51.63233135s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039223932 (104h27m51.635994945s) [noise=true , address=185.167.204.206:4052 , piece_id=AXXKU7XCFVOYI4KBLQ4P5JOCNVWTR27HDBB6LLCFMRFEKRE3VOQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039642416 (104h27m51.636413439s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.041029145 (104h27m51.637800171s) [noise=true , address=81.171.14.70:4047 , piece_id=DRMYHFSFKE6W754ZVGR4443E4IN3RSLIDF4YVS4JLRWTHJJSGNRA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.044680567 (104h27m51.64145159s) [noise=true , address=81.171.14.70:4037 , piece_id=U5Z5435XRWAWRIAS5HFPZI3YHGPRVWPRQB2BSOCLZDO3NGTMKO5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.050626247 (104h27m51.647397275s) [noise=true , address=185.167.204.206:4058 , piece_id=NEUNBF5MTH7ZNJ3OKD2SCMOSILZ7THBMO6EYNHZ4IQNIQ5T3IJ7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.063548076 (104h27m51.6603191s) [noise=true , address=185.167.204.205:4014 , piece_id=7JROEIOUUMNHTLLV3S4D3Q2RJ6JEZZBF6GHJI3BE635Z7NRRPPYA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.080832704 (104h27m51.677603718s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.088180011 (104h27m51.684951025s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.109755006 (104h27m51.706526034s) [noise=true , address=185.167.204.205:4035 , piece_id=PPE2DXQIF574R55F7BGW5SRV2MTAROPD2QERBVJ6IXKV4HMCIHFA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.117984644 (104h27m51.71475567s) [noise=true , address=185.167.204.205:4019 , piece_id=FXPKM5PNNZC5SEE3PBR7VPGX4Y644QCC6IBRJTF4PVRVUEC7XEQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.118587921 (104h27m51.715358933s) [noise=true , address=185.167.204.206:4040 , piece_id=WBPAL2X62XGHRZ6KPNSHAXNE4U5XDXYTESCTVBW6F4G4JAF5JALQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.127117193 (104h27m51.723888222s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.166283722 (104h27m51.763054748s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.472291315 (104h27m52.069062329s) [noise=true , address=81.171.14.70:4040 , piece_id=GIBXDZKK4H5WKV4AR2VAA6BWYNPN3AYOD66DN4WGWZPXKXHRRW5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.486254788 (104h27m52.083025819s) [noise=true , address=89.149.201.147:4013 , piece_id=X3IEGLOWJDQTXWRFFTEJGZFBD5TQJNURDIHP2RUGJE65YJUQAISQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.552368074 (104h27m52.149139108s) [noise=true , address=89.149.201.147:4023 , piece_id=JWOFVPWTRQ45HXXXKDRH2MDBKQL7MC6VNMBSLPBVJNDHDBE5XTAA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:56.435632635 (104h27m54.032403649s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:58.800969733 (104h27m56.397740758s) [noise=true , address=185.167.204.205:4033 , piece_id=O2SIBP4ZSONAV6MUS6IDXXNM7PDLJQBQBELUOUBXZOSKLDSIR7NA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:28:53.009605827 (104h28m50.606376886s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: read tcp 159.223.143.39:34090->89.149.201.147:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009693883 (104h28m50.606464943s) [noise=true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: read tcp 159.223.143.39:33010->89.149.201.147:4017: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009755582 (104h28m50.606526616s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 159.223.143.39:48940->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009829068 (104h28m50.606600111s) [seg_pos={0 0}, upload=116916]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:28:53.106470711 (104h28m50.703241735s) [upload=116916]: Done uploading segments... Mar 10 19:28:53.106515494 (104h28m50.703286524s) [upload=116916]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.156310609s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...X...................X......X............................] duration: 916.745939ms trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:31:44 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 916.771636ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...X...................X......X...........................] duration: 1.232876074s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:31:25 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.232904586s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X....X..........X...................................] duration: 1.046570501s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:31:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.046602709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 6 out of 60 is failed (threshold=3) [...........................X......X......XX.X...........X...] duration: 2.409203876s trace: https://jaeger.dev.storj.io/trace/3b4460442365c0e7 FAIL (2026-03-10 19:30:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389309232 (104h27m50.986080258s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389816473 (104h27m50.986587499s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.391931427 (104h27m50.988702464s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.394396629 (104h27m50.991167767s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.397263419 (104h27m50.994034452s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.399366031 (104h27m50.996137061s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.406080001 (104h27m51.002851053s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.409602698 (104h27m51.006373731s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.416502293 (104h27m51.013273329s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.420753887 (104h27m51.017524921s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.423262462 (104h27m51.020033496s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.823101372 (104h27m51.419872405s) [upload=116916]: Next returned nil segment Mar 10 19:27:53.823120446 (104h27m51.419891481s) [upload=116916]: Waiting for error group managing segments... Mar 10 19:27:54.000664652 (104h27m51.597435708s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.008012848 (104h27m51.604783892s) [noise=true , address=81.171.14.70:4029 , piece_id=4LYITJ27FYZX25WUL42RFJUX7VSDBA7CXJJASFKRAFGPXJYKM6QA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.011975049 (104h27m51.608746079s) [noise=true , address=81.171.14.70:4038 , piece_id=2HBQKDA5TGBX2P7NASC6NA4MWD7NVDUUG3VBAECSTG5TRDDR2I3Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013324482 (104h27m51.610095512s) [noise=true , address=185.167.204.205:4032 , piece_id=VXAHZM6L7462VFYIJFBL6RHMJAYOGY3EODRUOHWII4HKM255UGZA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013614745 (104h27m51.610385775s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.026623408 (104h27m51.623394435s) [noise=true , address=81.171.14.70:4033 , piece_id=LWM6DL6LLEPLGWK3Q3QIYLRYEJA4FWT7OUN5WPSRXPERNKFZKIIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.035560337 (104h27m51.63233135s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039223932 (104h27m51.635994945s) [noise=true , address=185.167.204.206:4052 , piece_id=AXXKU7XCFVOYI4KBLQ4P5JOCNVWTR27HDBB6LLCFMRFEKRE3VOQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039642416 (104h27m51.636413439s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.041029145 (104h27m51.637800171s) [noise=true , address=81.171.14.70:4047 , piece_id=DRMYHFSFKE6W754ZVGR4443E4IN3RSLIDF4YVS4JLRWTHJJSGNRA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.044680567 (104h27m51.64145159s) [noise=true , address=81.171.14.70:4037 , piece_id=U5Z5435XRWAWRIAS5HFPZI3YHGPRVWPRQB2BSOCLZDO3NGTMKO5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.050626247 (104h27m51.647397275s) [noise=true , address=185.167.204.206:4058 , piece_id=NEUNBF5MTH7ZNJ3OKD2SCMOSILZ7THBMO6EYNHZ4IQNIQ5T3IJ7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.063548076 (104h27m51.6603191s) [noise=true , address=185.167.204.205:4014 , piece_id=7JROEIOUUMNHTLLV3S4D3Q2RJ6JEZZBF6GHJI3BE635Z7NRRPPYA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.080832704 (104h27m51.677603718s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.088180011 (104h27m51.684951025s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.109755006 (104h27m51.706526034s) [noise=true , address=185.167.204.205:4035 , piece_id=PPE2DXQIF574R55F7BGW5SRV2MTAROPD2QERBVJ6IXKV4HMCIHFA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.117984644 (104h27m51.71475567s) [noise=true , address=185.167.204.205:4019 , piece_id=FXPKM5PNNZC5SEE3PBR7VPGX4Y644QCC6IBRJTF4PVRVUEC7XEQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.118587921 (104h27m51.715358933s) [noise=true , address=185.167.204.206:4040 , piece_id=WBPAL2X62XGHRZ6KPNSHAXNE4U5XDXYTESCTVBW6F4G4JAF5JALQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.127117193 (104h27m51.723888222s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.166283722 (104h27m51.763054748s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.472291315 (104h27m52.069062329s) [noise=true , address=81.171.14.70:4040 , piece_id=GIBXDZKK4H5WKV4AR2VAA6BWYNPN3AYOD66DN4WGWZPXKXHRRW5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.486254788 (104h27m52.083025819s) [noise=true , address=89.149.201.147:4013 , piece_id=X3IEGLOWJDQTXWRFFTEJGZFBD5TQJNURDIHP2RUGJE65YJUQAISQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.552368074 (104h27m52.149139108s) [noise=true , address=89.149.201.147:4023 , piece_id=JWOFVPWTRQ45HXXXKDRH2MDBKQL7MC6VNMBSLPBVJNDHDBE5XTAA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:56.435632635 (104h27m54.032403649s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:58.800969733 (104h27m56.397740758s) [noise=true , address=185.167.204.205:4033 , piece_id=O2SIBP4ZSONAV6MUS6IDXXNM7PDLJQBQBELUOUBXZOSKLDSIR7NA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:28:53.009605827 (104h28m50.606376886s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: read tcp 159.223.143.39:34090->89.149.201.147:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009693883 (104h28m50.606464943s) [noise=true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: read tcp 159.223.143.39:33010->89.149.201.147:4017: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009755582 (104h28m50.606526616s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 159.223.143.39:48940->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009829068 (104h28m50.606600111s) [seg_pos={0 0}, upload=116916]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:28:53.106470711 (104h28m50.703241735s) [upload=116916]: Done uploading segments... Mar 10 19:28:53.106515494 (104h28m50.703286524s) [upload=116916]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.409248795s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...X...................X......X..........................] duration: 1.126483038s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:30:32 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.126532431s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X....X..........X..................................] duration: 1.266660646s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:30:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.266774552s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 6 out of 60 is failed (threshold=3) [............................X......X......XX.X...........X..] duration: 2.058845484s trace: https://jaeger.dev.storj.io/trace/3b4460442365c0e7 FAIL (2026-03-10 19:29:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389309232 (104h27m50.986080258s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389816473 (104h27m50.986587499s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.391931427 (104h27m50.988702464s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.394396629 (104h27m50.991167767s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.397263419 (104h27m50.994034452s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.399366031 (104h27m50.996137061s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.406080001 (104h27m51.002851053s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.409602698 (104h27m51.006373731s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.416502293 (104h27m51.013273329s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.420753887 (104h27m51.017524921s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.423262462 (104h27m51.020033496s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.823101372 (104h27m51.419872405s) [upload=116916]: Next returned nil segment Mar 10 19:27:53.823120446 (104h27m51.419891481s) [upload=116916]: Waiting for error group managing segments... Mar 10 19:27:54.000664652 (104h27m51.597435708s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.008012848 (104h27m51.604783892s) [noise=true , address=81.171.14.70:4029 , piece_id=4LYITJ27FYZX25WUL42RFJUX7VSDBA7CXJJASFKRAFGPXJYKM6QA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.011975049 (104h27m51.608746079s) [noise=true , address=81.171.14.70:4038 , piece_id=2HBQKDA5TGBX2P7NASC6NA4MWD7NVDUUG3VBAECSTG5TRDDR2I3Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013324482 (104h27m51.610095512s) [noise=true , address=185.167.204.205:4032 , piece_id=VXAHZM6L7462VFYIJFBL6RHMJAYOGY3EODRUOHWII4HKM255UGZA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013614745 (104h27m51.610385775s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.026623408 (104h27m51.623394435s) [noise=true , address=81.171.14.70:4033 , piece_id=LWM6DL6LLEPLGWK3Q3QIYLRYEJA4FWT7OUN5WPSRXPERNKFZKIIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.035560337 (104h27m51.63233135s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039223932 (104h27m51.635994945s) [noise=true , address=185.167.204.206:4052 , piece_id=AXXKU7XCFVOYI4KBLQ4P5JOCNVWTR27HDBB6LLCFMRFEKRE3VOQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039642416 (104h27m51.636413439s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.041029145 (104h27m51.637800171s) [noise=true , address=81.171.14.70:4047 , piece_id=DRMYHFSFKE6W754ZVGR4443E4IN3RSLIDF4YVS4JLRWTHJJSGNRA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.044680567 (104h27m51.64145159s) [noise=true , address=81.171.14.70:4037 , piece_id=U5Z5435XRWAWRIAS5HFPZI3YHGPRVWPRQB2BSOCLZDO3NGTMKO5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.050626247 (104h27m51.647397275s) [noise=true , address=185.167.204.206:4058 , piece_id=NEUNBF5MTH7ZNJ3OKD2SCMOSILZ7THBMO6EYNHZ4IQNIQ5T3IJ7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.063548076 (104h27m51.6603191s) [noise=true , address=185.167.204.205:4014 , piece_id=7JROEIOUUMNHTLLV3S4D3Q2RJ6JEZZBF6GHJI3BE635Z7NRRPPYA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.080832704 (104h27m51.677603718s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.088180011 (104h27m51.684951025s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.109755006 (104h27m51.706526034s) [noise=true , address=185.167.204.205:4035 , piece_id=PPE2DXQIF574R55F7BGW5SRV2MTAROPD2QERBVJ6IXKV4HMCIHFA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.117984644 (104h27m51.71475567s) [noise=true , address=185.167.204.205:4019 , piece_id=FXPKM5PNNZC5SEE3PBR7VPGX4Y644QCC6IBRJTF4PVRVUEC7XEQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.118587921 (104h27m51.715358933s) [noise=true , address=185.167.204.206:4040 , piece_id=WBPAL2X62XGHRZ6KPNSHAXNE4U5XDXYTESCTVBW6F4G4JAF5JALQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.127117193 (104h27m51.723888222s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.166283722 (104h27m51.763054748s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.472291315 (104h27m52.069062329s) [noise=true , address=81.171.14.70:4040 , piece_id=GIBXDZKK4H5WKV4AR2VAA6BWYNPN3AYOD66DN4WGWZPXKXHRRW5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.486254788 (104h27m52.083025819s) [noise=true , address=89.149.201.147:4013 , piece_id=X3IEGLOWJDQTXWRFFTEJGZFBD5TQJNURDIHP2RUGJE65YJUQAISQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.552368074 (104h27m52.149139108s) [noise=true , address=89.149.201.147:4023 , piece_id=JWOFVPWTRQ45HXXXKDRH2MDBKQL7MC6VNMBSLPBVJNDHDBE5XTAA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:56.435632635 (104h27m54.032403649s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:58.800969733 (104h27m56.397740758s) [noise=true , address=185.167.204.205:4033 , piece_id=O2SIBP4ZSONAV6MUS6IDXXNM7PDLJQBQBELUOUBXZOSKLDSIR7NA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:28:53.009605827 (104h28m50.606376886s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: read tcp 159.223.143.39:34090->89.149.201.147:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009693883 (104h28m50.606464943s) [noise=true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: read tcp 159.223.143.39:33010->89.149.201.147:4017: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009755582 (104h28m50.606526616s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 159.223.143.39:48940->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009829068 (104h28m50.606600111s) [seg_pos={0 0}, upload=116916]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:28:53.106470711 (104h28m50.703241735s) [upload=116916]: Done uploading segments... Mar 10 19:28:53.106515494 (104h28m50.703286524s) [upload=116916]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.058884007s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X....X..........X.................................] duration: 1.248086764s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:29:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.248125879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X...X...................X......X.........................] duration: 1.143972241s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:29:26 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.144000131s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 6 out of 60 is failed (threshold=3) [.............................X......X......XX.X...........X.] duration: 10.460870451s trace: https://jaeger.dev.storj.io/trace/3b4460442365c0e7 FAIL (2026-03-10 19:29:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389309232 (104h27m50.986080258s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389816473 (104h27m50.986587499s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.391931427 (104h27m50.988702464s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.394396629 (104h27m50.991167767s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.397263419 (104h27m50.994034452s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.399366031 (104h27m50.996137061s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.406080001 (104h27m51.002851053s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.409602698 (104h27m51.006373731s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.416502293 (104h27m51.013273329s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.420753887 (104h27m51.017524921s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.423262462 (104h27m51.020033496s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.823101372 (104h27m51.419872405s) [upload=116916]: Next returned nil segment Mar 10 19:27:53.823120446 (104h27m51.419891481s) [upload=116916]: Waiting for error group managing segments... Mar 10 19:27:54.000664652 (104h27m51.597435708s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.008012848 (104h27m51.604783892s) [noise=true , address=81.171.14.70:4029 , piece_id=4LYITJ27FYZX25WUL42RFJUX7VSDBA7CXJJASFKRAFGPXJYKM6QA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.011975049 (104h27m51.608746079s) [noise=true , address=81.171.14.70:4038 , piece_id=2HBQKDA5TGBX2P7NASC6NA4MWD7NVDUUG3VBAECSTG5TRDDR2I3Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013324482 (104h27m51.610095512s) [noise=true , address=185.167.204.205:4032 , piece_id=VXAHZM6L7462VFYIJFBL6RHMJAYOGY3EODRUOHWII4HKM255UGZA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013614745 (104h27m51.610385775s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.026623408 (104h27m51.623394435s) [noise=true , address=81.171.14.70:4033 , piece_id=LWM6DL6LLEPLGWK3Q3QIYLRYEJA4FWT7OUN5WPSRXPERNKFZKIIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.035560337 (104h27m51.63233135s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039223932 (104h27m51.635994945s) [noise=true , address=185.167.204.206:4052 , piece_id=AXXKU7XCFVOYI4KBLQ4P5JOCNVWTR27HDBB6LLCFMRFEKRE3VOQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039642416 (104h27m51.636413439s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.041029145 (104h27m51.637800171s) [noise=true , address=81.171.14.70:4047 , piece_id=DRMYHFSFKE6W754ZVGR4443E4IN3RSLIDF4YVS4JLRWTHJJSGNRA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.044680567 (104h27m51.64145159s) [noise=true , address=81.171.14.70:4037 , piece_id=U5Z5435XRWAWRIAS5HFPZI3YHGPRVWPRQB2BSOCLZDO3NGTMKO5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.050626247 (104h27m51.647397275s) [noise=true , address=185.167.204.206:4058 , piece_id=NEUNBF5MTH7ZNJ3OKD2SCMOSILZ7THBMO6EYNHZ4IQNIQ5T3IJ7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.063548076 (104h27m51.6603191s) [noise=true , address=185.167.204.205:4014 , piece_id=7JROEIOUUMNHTLLV3S4D3Q2RJ6JEZZBF6GHJI3BE635Z7NRRPPYA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.080832704 (104h27m51.677603718s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.088180011 (104h27m51.684951025s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.109755006 (104h27m51.706526034s) [noise=true , address=185.167.204.205:4035 , piece_id=PPE2DXQIF574R55F7BGW5SRV2MTAROPD2QERBVJ6IXKV4HMCIHFA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.117984644 (104h27m51.71475567s) [noise=true , address=185.167.204.205:4019 , piece_id=FXPKM5PNNZC5SEE3PBR7VPGX4Y644QCC6IBRJTF4PVRVUEC7XEQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.118587921 (104h27m51.715358933s) [noise=true , address=185.167.204.206:4040 , piece_id=WBPAL2X62XGHRZ6KPNSHAXNE4U5XDXYTESCTVBW6F4G4JAF5JALQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.127117193 (104h27m51.723888222s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.166283722 (104h27m51.763054748s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.472291315 (104h27m52.069062329s) [noise=true , address=81.171.14.70:4040 , piece_id=GIBXDZKK4H5WKV4AR2VAA6BWYNPN3AYOD66DN4WGWZPXKXHRRW5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.486254788 (104h27m52.083025819s) [noise=true , address=89.149.201.147:4013 , piece_id=X3IEGLOWJDQTXWRFFTEJGZFBD5TQJNURDIHP2RUGJE65YJUQAISQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.552368074 (104h27m52.149139108s) [noise=true , address=89.149.201.147:4023 , piece_id=JWOFVPWTRQ45HXXXKDRH2MDBKQL7MC6VNMBSLPBVJNDHDBE5XTAA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:56.435632635 (104h27m54.032403649s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:58.800969733 (104h27m56.397740758s) [noise=true , address=185.167.204.205:4033 , piece_id=O2SIBP4ZSONAV6MUS6IDXXNM7PDLJQBQBELUOUBXZOSKLDSIR7NA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:28:53.009605827 (104h28m50.606376886s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: read tcp 159.223.143.39:34090->89.149.201.147:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009693883 (104h28m50.606464943s) [noise=true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: read tcp 159.223.143.39:33010->89.149.201.147:4017: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009755582 (104h28m50.606526616s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 159.223.143.39:48940->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009829068 (104h28m50.606600111s) [seg_pos={0 0}, upload=116916]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:28:53.106470711 (104h28m50.703241735s) [upload=116916]: Done uploading segments... Mar 10 19:28:53.106515494 (104h28m50.703286524s) [upload=116916]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 10.460891673s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 6 out of 60 is failed (threshold=3) [..............................X......X......XX.X...........X] duration: 1m0.100077576s trace: https://jaeger.dev.storj.io/trace/3b4460442365c0e7 FAIL (2026-03-10 19:28:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389309232 (104h27m50.986080258s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389816473 (104h27m50.986587499s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.391931427 (104h27m50.988702464s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.394396629 (104h27m50.991167767s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.397263419 (104h27m50.994034452s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.399366031 (104h27m50.996137061s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.406080001 (104h27m51.002851053s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.409602698 (104h27m51.006373731s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.416502293 (104h27m51.013273329s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.420753887 (104h27m51.017524921s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.423262462 (104h27m51.020033496s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.823101372 (104h27m51.419872405s) [upload=116916]: Next returned nil segment Mar 10 19:27:53.823120446 (104h27m51.419891481s) [upload=116916]: Waiting for error group managing segments... Mar 10 19:27:54.000664652 (104h27m51.597435708s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.008012848 (104h27m51.604783892s) [noise=true , address=81.171.14.70:4029 , piece_id=4LYITJ27FYZX25WUL42RFJUX7VSDBA7CXJJASFKRAFGPXJYKM6QA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.011975049 (104h27m51.608746079s) [noise=true , address=81.171.14.70:4038 , piece_id=2HBQKDA5TGBX2P7NASC6NA4MWD7NVDUUG3VBAECSTG5TRDDR2I3Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013324482 (104h27m51.610095512s) [noise=true , address=185.167.204.205:4032 , piece_id=VXAHZM6L7462VFYIJFBL6RHMJAYOGY3EODRUOHWII4HKM255UGZA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013614745 (104h27m51.610385775s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.026623408 (104h27m51.623394435s) [noise=true , address=81.171.14.70:4033 , piece_id=LWM6DL6LLEPLGWK3Q3QIYLRYEJA4FWT7OUN5WPSRXPERNKFZKIIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.035560337 (104h27m51.63233135s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039223932 (104h27m51.635994945s) [noise=true , address=185.167.204.206:4052 , piece_id=AXXKU7XCFVOYI4KBLQ4P5JOCNVWTR27HDBB6LLCFMRFEKRE3VOQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039642416 (104h27m51.636413439s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.041029145 (104h27m51.637800171s) [noise=true , address=81.171.14.70:4047 , piece_id=DRMYHFSFKE6W754ZVGR4443E4IN3RSLIDF4YVS4JLRWTHJJSGNRA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.044680567 (104h27m51.64145159s) [noise=true , address=81.171.14.70:4037 , piece_id=U5Z5435XRWAWRIAS5HFPZI3YHGPRVWPRQB2BSOCLZDO3NGTMKO5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.050626247 (104h27m51.647397275s) [noise=true , address=185.167.204.206:4058 , piece_id=NEUNBF5MTH7ZNJ3OKD2SCMOSILZ7THBMO6EYNHZ4IQNIQ5T3IJ7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.063548076 (104h27m51.6603191s) [noise=true , address=185.167.204.205:4014 , piece_id=7JROEIOUUMNHTLLV3S4D3Q2RJ6JEZZBF6GHJI3BE635Z7NRRPPYA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.080832704 (104h27m51.677603718s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.088180011 (104h27m51.684951025s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.109755006 (104h27m51.706526034s) [noise=true , address=185.167.204.205:4035 , piece_id=PPE2DXQIF574R55F7BGW5SRV2MTAROPD2QERBVJ6IXKV4HMCIHFA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.117984644 (104h27m51.71475567s) [noise=true , address=185.167.204.205:4019 , piece_id=FXPKM5PNNZC5SEE3PBR7VPGX4Y644QCC6IBRJTF4PVRVUEC7XEQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.118587921 (104h27m51.715358933s) [noise=true , address=185.167.204.206:4040 , piece_id=WBPAL2X62XGHRZ6KPNSHAXNE4U5XDXYTESCTVBW6F4G4JAF5JALQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.127117193 (104h27m51.723888222s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.166283722 (104h27m51.763054748s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.472291315 (104h27m52.069062329s) [noise=true , address=81.171.14.70:4040 , piece_id=GIBXDZKK4H5WKV4AR2VAA6BWYNPN3AYOD66DN4WGWZPXKXHRRW5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.486254788 (104h27m52.083025819s) [noise=true , address=89.149.201.147:4013 , piece_id=X3IEGLOWJDQTXWRFFTEJGZFBD5TQJNURDIHP2RUGJE65YJUQAISQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.552368074 (104h27m52.149139108s) [noise=true , address=89.149.201.147:4023 , piece_id=JWOFVPWTRQ45HXXXKDRH2MDBKQL7MC6VNMBSLPBVJNDHDBE5XTAA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:56.435632635 (104h27m54.032403649s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:58.800969733 (104h27m56.397740758s) [noise=true , address=185.167.204.205:4033 , piece_id=O2SIBP4ZSONAV6MUS6IDXXNM7PDLJQBQBELUOUBXZOSKLDSIR7NA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:28:53.009605827 (104h28m50.606376886s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: read tcp 159.223.143.39:34090->89.149.201.147:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009693883 (104h28m50.606464943s) [noise=true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: read tcp 159.223.143.39:33010->89.149.201.147:4017: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009755582 (104h28m50.606526616s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 159.223.143.39:48940->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009829068 (104h28m50.606600111s) [seg_pos={0 0}, upload=116916]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:28:53.106470711 (104h28m50.703241735s) [upload=116916]: Done uploading segments... Mar 10 19:28:53.106515494 (104h28m50.703286524s) [upload=116916]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 1m0.100103051s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X...X...X...................X......X........................] duration: 1.089333441s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:28:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.089368706s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X....X..........X................................] duration: 1.297920894s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:28:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.297970142s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X....X..........X...............................] duration: 1.167480843s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:28:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.167508671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X...X...X...................X......X.......................] duration: 1.212047323s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:28:03 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.212076449s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 613.779562ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:27:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 613.811776ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X....X..........X..............................] duration: 1.347438113s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:27:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.34747784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...X...X...................X......X......................] duration: 1.218900486s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:27:34 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.218927009s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.............X....X..........X.............................] duration: 1.273706035s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:27:24 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.27373006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X...X...X...................X......X.....................] duration: 1.097981522s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:27:10 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.0980123s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)