Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.046014504s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:39:15 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.046109103s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 472.947958ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:39:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 472.9743ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.026487203s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:38:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.026509932s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 814.283141ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:38:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 814.308246ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 445.05276ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:38:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 445.08009ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.916565484s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:38:18 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.916586506s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 381.195868ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:38:05 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 381.236561ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 598.282279ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:37:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 598.310706ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 391.578596ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:37:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 391.603397ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.742437181s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:37:11 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.742466583s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 391.204464ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:37:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 391.226274ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 479.78658ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:37:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 479.81069ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 513.735042ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:36:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 513.771775ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 345.513584ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:36:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 345.546036ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.923204036s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:36:06 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.923231662s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 570.945229ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:35:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 570.970216ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 213.074494ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:35:17 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 213.119462ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.913490361s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:35:11 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.913518426s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 672.062052ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:35:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 672.144234ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.786861919s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:34:40 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.786884656s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 651.085811ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:34:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 651.106182ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 435.389638ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:34:05 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 435.404431ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 541.594053ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:33:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 541.626978ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.934978681s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:33:44 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.935007486s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 743.196099ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:33:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 743.23777ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.917939621s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:33:24 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.917965778s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 426.674096ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:33:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 426.718236ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.136190702s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:32:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.136218523s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 601.283633ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:32:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 601.313609ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 448.122434ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:32:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 448.144569ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.800093243s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:32:25 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.800122514s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 524.528491ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:32:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 524.552535ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 340.888195ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:32:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 340.914215ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 512.794963ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:31:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 512.820585ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.906276662s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:31:38 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.906301492s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 574.83442ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:31:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 574.884185ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 604.618012ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:30:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 604.64043ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.836479475s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:30:55 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.836502924s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 419.893317ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:30:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 419.988613ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 438.795505ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:30:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 438.829107ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.738404214s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:30:07 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.738437168s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 397.249257ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:29:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 397.281902ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 353.627312ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:29:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 353.656056ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 370.923846ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:29:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 370.947366ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 426.866455ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:29:12 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 426.90138ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 406.048188ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:28:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 406.075351ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.912153248s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:28:54 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.912178829s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 410.968603ms trace: https://jaeger.dev.storj.io/trace/2ff3cb61697ef4b6 FAIL (2025-12-02 18:28:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (512h12m6.565207247s) [upload=533150]: Starting upload Dec 2 17:58:41.797600373 (512h12m6.565283807s) [upload=533150]: Done starting upload Dec 2 17:58:41.797703493 (512h12m6.565386907s) [upload=533150]: Uploading segments... Dec 2 17:58:41.797922579 (512h12m6.565606005s) [upload=533150]: Got next segment. Inline: false Dec 2 17:58:41.797975464 (512h12m6.56565889s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:41.797980961 (512h12m6.565664386s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 2 17:58:45.826641704 (512h12m10.594325119s) [seg_pos={0 0}, upload=533150]: Begin upload segment... Dec 2 17:58:45.826704608 (512h12m10.594388034s) [seg_pos={0 0}, upload=533150]: Done begin upload segment. Dec 2 17:58:45.826736929 (512h12m10.594420354s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827113477 (512h12m10.594796905s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.827790378 (512h12m10.595473806s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828185978 (512h12m10.595869406s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828333113 (512h12m10.596016544s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828681485 (512h12m10.596364914s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.828853063 (512h12m10.596536492s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829056243 (512h12m10.596739671s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829243823 (512h12m10.596927248s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829468022 (512h12m10.597151447s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829620763 (512h12m10.597304195s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.829919326 (512h12m10.597602755s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830074947 (512h12m10.597758373s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830364126 (512h12m10.598047551s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830427732 (512h12m10.598111161s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.830818895 (512h12m10.59850232s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Uploading piece... Dec 2 17:58:45.929763669 (512h12m10.697447094s) [noise=true , address=145.79.227.34:4906 , piece_id=NMV35C5H3Q53ZVHJPEQQDTQY4UB7NRYTFD5SC7N7ANYG7IQM2CPQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.929826388 (512h12m10.697509814s) [upload=533150]: Next returned nil segment Dec 2 17:58:45.929841528 (512h12m10.69752496s) [upload=533150]: Waiting for error group managing segments... Dec 2 17:58:45.931514752 (512h12m10.699198177s) [noise=true , address=145.79.227.17:4971 , piece_id=5YBOSDBLQ5WAYAQQ22YWNKXV2IR3UOHYTNM4RW4QYIK5QOJWRVZQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932024553 (512h12m10.699707982s) [noise=true , address=77.90.31.14:4663 , piece_id=VZFCAJEFATYWIST6CFZIYGE646HF2B43O6OBI7ARLYUFEC7253AA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.932805477 (512h12m10.700488914s) [noise=true , address=145.79.227.14:4837 , piece_id=BPGESEZORPQPUSXHWQEH6E6NT5L4IDNSAOX4L7UZPRBTXYITRCKA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933099890 (512h12m10.700783316s) [noise=true , address=77.90.31.11:4554 , piece_id=374VOUS3GHZVX64PSHY53DP426UBOS6UAHWHK6S4GMNGAPMFHN2A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.933627663 (512h12m10.701311099s) [noise=true , address=145.79.227.9:4367 , piece_id=HERJJZCHIPKNVPFJVXNK3Y4KKT6UY64NLLKY7O3N6CQUJBCATWXA, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935225037 (512h12m10.702908483s) [noise=true , address=145.79.227.10:4401 , piece_id=QT7N453UNLOQVAPXNDMNH5S7FDP5SJBU6YTMTGTIPWMDA2CXP7QQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.935668414 (512h12m10.703351848s) [noise=true , address=145.79.227.33:4871 , piece_id=UUJBXXASIWHFJBADPBOTP6R2RHVM75BVJDYNM724VN6OGA3TB5IQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936585494 (512h12m10.704268924s) [noise=true , address=145.79.227.30:4761 , piece_id=HQ6YY6D6ULP5YRPAI3SPCDGOUI7O5BVHPSCOB44VISN3RK4KP52Q, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.936992930 (512h12m10.704676359s) [noise=true , address=145.79.227.38:4978 , piece_id=WX52RITMP6VKVJVDZAQ3LL6NH37CTXXSDW7NKSEKTMQKLXJ2YPAQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.938169624 (512h12m10.70585305s) [noise=true , address=145.79.227.15:4874 , piece_id=VMF6DLYNWVI6AFQGZPHC6ASKC7JULVPFJBAT7LFHA2JECWKTXRWQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.939839291 (512h12m10.707522721s) [noise=true , address=145.79.227.27:4689 , piece_id=SW6DMTAZP3DRBEU2VP2HL2Z6NTEH57GES7BG3AYNPWSGXIPQSR6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941144675 (512h12m10.708828104s) [noise=true , address=145.79.227.28:4727 , piece_id=YB5VUK64I33MEQQFLAZDQ7U25KCWFQLP2HARRB67BYYV76P26NNQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: <nil> Dec 2 17:58:45.941166602 (512h12m10.708850037s) [seg_pos={0 0}, upload=533150]: Segment reached optimal threshold of 13 pieces. Dec 2 17:58:45.941420693 (512h12m10.70910413s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38284->77.90.31.16:4734: use of closed network connection Dec 2 17:58:45.941433615 (512h12m10.709117052s) [noise=true , address=77.90.31.16:4734 , piece_id=OLZ7LESLE7ZSSJECIWPRVPS4KAHLRKAIAQT7Q44A2ASHAOFERYMQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941790383 (512h12m10.709473818s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.941800244 (512h12m10.709483679s) [noise=true , address=77.90.31.18:4806 , piece_id=6ZY5B6G6UYZ6ROUZCEUFFQ2P2A42CULO4CNWPNEOS6T6XS7EDO6A, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.941993661 (512h12m10.709677095s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 2 17:58:45.942004595 (512h12m10.709688033s) [noise=true , address=145.79.227.25:4618 , piece_id=XAWU77AM7JIBMX7BINPHJASZ63ZIBDLLN74D4GC53GWJLR6RH2DQ, seg_pos={0 0}, upload=533150]: Long tail cancellation due to optimal threshold Dec 2 17:58:45.942036759 (512h12m10.709720188s) [seg_pos={0 0}, upload=533150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 2 17:58:45.942132787 (512h12m10.709816219s) [upload=533150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 2 17:58:45.942138898 (512h12m10.709822327s) [upload=533150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:59:18.459127944 (512h12m43.226811375s) [upload=533150]: Done uploading segments... Dec 2 17:59:18.459220103 (512h12m43.226903534s) [upload=533150]: Upload finished. err: metaclient: context deadline exceeded Duration: 410.99416ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 390.690301ms trace: https://jaeger.dev.storj.io/trace/6e74a775ed8fdecd FAIL (2025-12-02 18:28:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 2 17:58:29.492296357 (512h11m54.259979784s) [upload=533148]: Starting upload Dec 2 17:58:29.492428778 (512h11m54.260112196s) [upload=533148]: Done starting upload Dec 2 17:58:29.493319189 (512h11m54.2610026s) [upload=533148]: Uploading segments... Dec 2 17:58:29.493401893 (512h11m54.261085318s) [upload=533148]: Got next segment. Inline: true Dec 2 17:58:29.493408011 (512h11m54.261091436s) [upload=533148]: Waiting for error group managing segments... Dec 2 17:58:29.493462867 (512h11m54.261146278s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 2 17:58:29.493468588 (512h11m54.261152015s) [upload=533148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 2 17:58:29.493473609 (512h11m54.261157044s) [upload=533148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 2 17:58:29.552401870 (512h11m54.320085306s) [upload=533148]: Done uploading segments... Dec 2 17:58:29.552428076 (512h11m54.320111505s) [upload=533148]: Upload finished. err: <nil> Duration:* 390.718894ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.779928551s trace: https://jaeger.dev.storj.io/trace/744490adeb12ced5 FAIL (2025-12-02 18:28:08 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.779953912s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)