Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.024328316s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:51:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.024359947s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.527038867s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:50:57 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.527065526s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.00819163s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:49:45 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.008214621s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.087977927s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:48:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.087996743s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.024342654s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:48:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.024373513s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.441223576s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:47:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.441249015s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.022337922s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:46:24 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.02242921s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.98121174s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:45:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 1.981237686s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.482232041s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:44:35 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.482254346s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.980251287s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:43:52 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 1.980274582s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.440063701s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:43:37 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.440090169s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.42003309s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:42:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.420057942s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 385.600705ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:41:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 385.639102ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 410.048045ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:41:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 410.075399ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.411948887s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:41:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.411973145s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 301.54453ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:40:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 301.572726ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.235964627s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:40:34 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.23599125s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.474414917s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:40:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.474439542s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 298.448399ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:39:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 298.481211ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.484445795s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:39:52 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.484474061s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 322.262428ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:38:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 322.292148ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.427568862s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:38:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.427584121s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 264.158431ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:38:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 264.1903ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.406132865s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:37:34 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.406159525s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 304.56242ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:37:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 304.587389ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 245.839929ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:37:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 245.866364ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.572414834s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:36:45 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.572451312s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.634664057s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:36:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.634703542s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 355.968542ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:36:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 355.995863ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 313.192987ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:35:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 313.215889ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.470546067s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:35:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.470571127s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 237.070664ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:34:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 237.105016ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 319.134706ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:34:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 319.158364ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.468660385s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:34:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.468685186s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 347.558399ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:33:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 347.584682ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.440104204s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:33:04 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.440131691s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 292.16717ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:32:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 292.1949ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.552801499s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:31:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.552824838s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.504101179s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:31:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.504126238s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 295.75035ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:31:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 295.841907ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 301.545233ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:30:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 301.561641ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.393743935s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:30:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.393785663s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.999448061s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:30:10 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 1.999476511s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 250.375956ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:29:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 250.399779ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.497114943s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:29:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.497143675s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 368.406766ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:28:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 368.433338ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.454756423s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:28:46 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 2.454854495s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 343.427038ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:28:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 343.517709ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.996895769s trace: https://jaeger.dev.storj.io/trace/2978a773c720f2e4 FAIL (2025-11-20 17:28:10 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=232637]: Uploading piece... Nov 20 17:25:10.582877896 (223h38m35.350561328s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.584580409 (223h38m35.352263843s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.586378791 (223h38m35.354062219s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.588641711 (223h38m35.356325142s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.591403897 (223h38m35.359087325s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.593348617 (223h38m35.361032051s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.595567737 (223h38m35.363251162s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.597504768 (223h38m35.365188202s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.599544427 (223h38m35.367227861s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.600434395 (223h38m35.36811783s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Uploading piece... Nov 20 17:25:10.992513100 (223h38m35.760196525s) [upload=232637]: Next returned nil segment Nov 20 17:25:10.992541848 (223h38m35.760225277s) [upload=232637]: Waiting for error group managing segments... Nov 20 17:25:11.164544942 (223h38m35.932228366s) [noise=true , address=89.149.201.147:4022 , piece_id=4NQ4J3UJAABVO3SFMRORPXBTUGYXZMJNUWWTAKFIH3ZD5EVF3UXQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.172767990 (223h38m35.940451418s) [noise=true , address=81.171.14.70:4031 , piece_id=SHBKH5K4C7T7D7XMAAD6FDA36VCEJXQILDGEVNJZLTMD6EXLBL5A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182086404 (223h38m35.949769833s) [noise=true , address=185.167.204.205:4015 , piece_id=ZC3TZGH2N7NHRNMFGG2SVOS5UGTD4CY4OCLCQL2EEIN6RHEMQKIQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.182810476 (223h38m35.950493902s) [noise=true , address=81.171.14.70:4043 , piece_id=6474UQDNSX4UVDXJWZKSDBB5T47HGTYJZMJNTFK3PEQBR2KAY7RA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.183742078 (223h38m35.951425504s) [noise=true , address=185.167.204.206:4053 , piece_id=AIF5XPKJPTGUA7ENKUUTPPOKQCECEKQPPGEMAUOT5QAISCVMO22Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.184529255 (223h38m35.952212684s) [noise=true , address=81.171.14.70:4044 , piece_id=RICD4ZMMWVGLS6UTFKKZFUNKE2LI35QQ6VF4FYRMLDUMQDOQCCXA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.185123500 (223h38m35.952806926s) [noise=true , address=185.167.204.206:4066 , piece_id=H52XDJBLWKNNCE25VSBRNUH6BMW3JPBOFGQSEKF4CI3HBKCQSJZQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.186964052 (223h38m35.954647478s) [noise=true , address=185.167.204.205:4028 , piece_id=FVBJ6WVHKSN43OVNMH66KPBREGS2EWGD2EWCVJQX2WVMI27MPC7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.191410051 (223h38m35.959093477s) [noise=true , address=185.167.204.206:4054 , piece_id=LCBZUECG7VIQOE3X4VT6AT4HEQEEOTHEFBPSRXG7NUJC4PCGQ72Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.194616300 (223h38m35.962299725s) [noise=true , address=89.149.201.147:4001 , piece_id=7YTUOPW4BA4KDNJAXDHNUENBDIXNGZZW6576XQD5D3ISQXKBTYTQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.195124666 (223h38m35.962808091s) [noise=true , address=81.171.14.70:4027 , piece_id=OOS4I4CDV2TN42MWMHZBRCQWXGYXLHJTTUONDMTAHSQJKKPROYJA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.196224298 (223h38m35.963907724s) [noise=true , address=89.149.201.147:4005 , piece_id=U64XSFTL2TDVWNNOUQ6ULUGBPNU5XGQ6SI3HU7ZP6PPLZ7PZVRQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.198703124 (223h38m35.966386601s) [noise=true , address=185.167.204.206:4047 , piece_id=YZFK5OW7TWWIJ6FLA2XMAU3YURPKT2U5FGZYH6YLRNHUUDIV5N7A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.201965022 (223h38m35.969648447s) [noise=true , address=185.167.204.205:4034 , piece_id=MPKSGXYASP3YGAXWHL4PLGA7PJX56ELPS6NPJQIKLQN3GOYAUFDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.202483607 (223h38m35.970167033s) [noise=true , address=185.167.204.205:4002 , piece_id=27KMJOJSSJTIHEO5ZKY66GIG66P5QAESU4HHHLTI6KV2WZCFRWYQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.211202242 (223h38m35.978885672s) [noise=true , address=185.167.204.205:4036 , piece_id=LZSPRFBBIJPCW3IWHNW3SNKX4YTW7RCXTOOLE5SBGR2Z4WJHVCZA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.216399102 (223h38m35.984082529s) [noise=true , address=185.167.204.206:4046 , piece_id=VY3VEVR5VIHEH3MWJKDYS2UM62EP7ELZU3JI346NDESEORJIPZDQ, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.226445839 (223h38m35.994129265s) [noise=true , address=185.167.204.205:4026 , piece_id=YVFA5OFDVPAN3Q7GP7NIO6MC2N47G2MEWLJ7RVJEIP7UBHBYD5NA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240411560 (223h38m36.008094989s) [noise=true , address=185.167.204.206:4065 , piece_id=UHVHSM2N5H73YGCYEJ26JWO6IZZ76JTTYWM5ZZZZ2HMERRK7VDWA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.240958840 (223h38m36.008642266s) [noise=true , address=81.171.14.70:4047 , piece_id=D46YVEDXW2RT6OUYO73PHKREJF55SOCRWX6AMJ6RUWUC7G67426A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.241565965 (223h38m36.009249392s) [noise=true , address=81.171.14.70:4036 , piece_id=RNDJKD3UBSBBCUSY6AMAJ46JA4FPSOOQMWP247A3CGPMMTZXWQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242175176 (223h38m36.009858606s) [noise=true , address=81.171.14.70:4028 , piece_id=HEZAKOJDOQEBIFDRGYIXSCUWLPKUBTA6MRKQPR6665ZQXEPVIGQA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.242769247 (223h38m36.010452683s) [noise=true , address=81.171.14.70:4030 , piece_id=QAOOJJY2QL6NZQ3DN4NG5MNQCLPFBEZ73BJ4ESWPHMMHFJZV3I6Q, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.243245042 (223h38m36.010928472s) [noise=true , address=81.171.14.70:4029 , piece_id=XJF5EGWPXCL3PXV6WXEL46JAHMHZQGGR4AU5IOUNQLBG4C3BGQUA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.247956007 (223h38m36.015639437s) [noise=true , address=81.171.14.70:4048 , piece_id=KUYGLZGDWTEMEG637NPTQCKAEOAU42SCT4QKIGTAFTIVTUTMUV6A, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251384119 (223h38m36.019067547s) [noise=true , address=81.171.14.70:4025 , piece_id=EZNZI3X3CQI5B6ZMKVZABPZ5IZHVVS722EXVLWRHEF625CZFGORA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: <nil> Nov 20 17:25:11.251410363 (223h38m36.019093791s) [seg_pos={0 0}, upload=232637]: Segment reached optimal threshold of 26 pieces. Nov 20 17:25:11.251626678 (223h38m36.019310112s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.251642292 (223h38m36.019325718s) [noise=true , address=89.149.201.147:4018 , piece_id=EF6O3MQ6T6Y6SZLNBTHJGH4U52JOUUIT5FJSRBBZB3R3XRH7MDKA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252013645 (223h38m36.019697078s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB): protocol: expected piece hash; context canceled; EOF Nov 20 17:25:11.252022652 (223h38m36.019706079s) [noise=true , address=81.171.14.70:4038 , piece_id=GCUUKZQY36F6XWCXJGHP6SSK5OKAH47HLPNBC76GVNNQID4DVLDA, seg_pos={0 0}, upload=232637]: Long tail cancellation due to optimal threshold Nov 20 17:25:11.252045137 (223h38m36.019728563s) [seg_pos={0 0}, upload=232637]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 20 17:25:11.252321599 (223h38m36.020005025s) [upload=232637]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 20 17:25:11.252330210 (223h38m36.020013635s) [upload=232637]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 17:25:11.439953049 (223h38m36.207636477s) [upload=232637]: Done uploading segments... Nov 20 17:25:11.439975748 (223h38m36.207659185s) [upload=232637]: Upload finished. err: <nil> Duration: 1.997060966s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 340.194019ms trace: https://jaeger.dev.storj.io/trace/4c3f0bf095c1ee35 FAIL (2025-11-20 17:28:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 20 16:59:28.751814362 (223h12m53.519497788s) [upload=232198]: Starting upload Nov 20 16:59:28.751887267 (223h12m53.519570685s) [upload=232198]: Done starting upload Nov 20 16:59:28.751969314 (223h12m53.519652729s) [upload=232198]: Uploading segments... Nov 20 16:59:28.752027711 (223h12m53.519711145s) [upload=232198]: Got next segment. Inline: true Nov 20 16:59:28.752032755 (223h12m53.51971618s) [upload=232198]: Waiting for error group managing segments... Nov 20 16:59:28.752093514 (223h12m53.519776939s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 20 16:59:28.752098565 (223h12m53.51978199s) [upload=232198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 20 16:59:28.752101755 (223h12m53.519785179s) [upload=232198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 20 16:59:58.720156033 (223h13m23.487839467s) [upload=232198]: Done uploading segments... Nov 20 16:59:58.720221444 (223h13m23.487904872s) [upload=232198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 340.224915ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)