Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 320.565732ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:15:28 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 320.592197ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 437.701937ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:15:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 437.725922ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 520.525358ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:14:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 520.554254ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 219.672751ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:14:53 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 219.694982ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 382.014909ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:14:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 382.042908ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.820731779s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:14:28 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.820755121s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 220.095669ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:14:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 220.120273ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.89469422s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:14:01 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.894736928s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 376.721347ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:13:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 376.744704ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 330.864054ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:13:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 330.892423ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 463.561991ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:12:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 463.583785ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.89670172s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:12:58 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.896798609s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 208.024983ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:12:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 208.118196ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.845316553s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:12:20 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.84541256s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 301.425477ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:12:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 301.53122ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 532.413773ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:11:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 532.44012ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 215.116127ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:11:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 215.140697ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 520.795184ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:11:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 520.811245ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.942149121s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:11:08 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.942177232s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 214.872945ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:10:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 214.901948ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.959968466s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:10:01 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.959998492s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 408.9886ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:09:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 409.038818ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 311.762945ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:09:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 311.793124ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 610.475134ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:09:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 610.499373ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.936861912s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:09:04 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.936965974s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 414.716934ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:08:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 414.744374ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.026343968s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:08:32 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.026369067s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 333.176775ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:08:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 333.220743ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 545.929777ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:08:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 545.954459ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 359.533969ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:07:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 359.561016ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 208.7614ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:07:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 208.791039ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.216779753s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:07:23 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.21681226s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 534.202981ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:07:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 534.24641ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.9835794s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:07:04 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.983619787s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 198.072864ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:06:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 198.10036ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.95424446s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:05:57 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.954275252s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 516.460694ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:05:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 516.482972ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 225.158261ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:05:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 225.206788ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 530.53212ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:05:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 530.559282ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.726224156s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:05:38 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.726327561s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.86493538s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:04:51 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.864964629s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 310.379132ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:04:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 310.40887ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 537.346616ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:04:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 537.373803ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.085179385s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:04:29 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.085288255s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 503.908139ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:04:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 503.937555ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 222.111507ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:04:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 222.149141ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 531.356345ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:03:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 531.382739ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 394.943399ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:03:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 394.962067ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.761594485s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:03:17 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.761628111s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 230.21495ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:03:02 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 230.311168ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)