Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 604.84297ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:27:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 604.872483ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 301.15217ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:27:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 301.179264ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.836034678s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:27:28 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.836058577s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 533.545902ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:27:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 533.571527ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.733579352s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:27:12 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.733617401s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 322.204851ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:26:57 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 322.24202ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.881835028s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:26:13 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.881864625s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 401.359504ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:26:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 401.381734ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 625.557422ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:26:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 625.646423ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 313.202668ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:25:46 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 313.296401ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.672394978s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:25:39 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.672423196s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 492.374563ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:25:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 492.39726ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 298.773516ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:25:17 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 298.804614ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.818767412s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:25:12 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.818790249s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 315.563409ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:24:57 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 315.602987ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 382.377938ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:24:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 382.403221ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 490.997788ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:24:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 491.030423ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 245.284326ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:24:22 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 245.312701ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.823091902s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:24:05 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.82311702s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 318.232583ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:23:28 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 318.259772ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 501.90191ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:23:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 501.957479ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.780055471s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:23:19 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.78008644s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.776530768s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:22:31 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.776553728s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 316.158088ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:22:25 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 316.220175ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 493.970444ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:22:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 493.996365ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.953253288s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:21:57 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.953297033s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 494.622902ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:21:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 494.645793ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 218.00636ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:21:17 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 218.036175ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 539.483501ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:21:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 539.513461ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 216.794494ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:20:59 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 216.828763ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.817017933s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:20:54 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.81705676s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.034683349s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:20:35 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.034710692s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 455.677658ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:19:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 455.705565ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 346.032407ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:19:46 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 346.061309ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.772943488s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:19:32 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.772972066s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 328.417133ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:19:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 328.462477ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 211.885543ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:18:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 211.915161ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 476.603453ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:18:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 476.633226ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.749753151s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:18:45 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.749781832s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 344.789733ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:18:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 344.816921ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 531.635896ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:18:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 531.65707ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.782675782s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:18:01 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.782711247s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 474.182674ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:17:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 474.211704ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 198.933868ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:17:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 199.023791ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.616574749s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:17:13 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.616604124s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 530.148176ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:16:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 530.212634ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.671815813s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:16:41 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.671841378s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 326.36169ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:16:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 326.380646ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 643.162765ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:15:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 643.190391ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.862461526s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:15:37 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.862496178s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)