Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 845.721707ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:17:19 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 845.737805ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 904.635394ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:16:40 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 904.727903ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 848.040576ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:16:03 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 848.075578ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 850.618632ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:14:54 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 850.709022ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 924.863091ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:14:37 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 924.927216ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 820.909296ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:14:06 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 820.946843ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 840.19173ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:13:35 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 840.219597ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.477467568s trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:12:24 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.477495463s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 902.910586ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:11:39 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 902.937831ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 957.485234ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:10:44 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 957.517795ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 928.406245ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:09:53 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 928.438776ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 905.468531ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:09:00 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 905.512287ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 873.704594ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:08:13 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 873.739664ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.123285004s trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:07:12 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.123322692s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.001461662s trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:06:18 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.001513342s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 954.392127ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:05:20 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 954.419641ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 938.384424ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:04:07 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 938.488432ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 859.199236ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:03:27 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 859.234434ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.045316542s trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:02:32 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.045344871s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.092191146s trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:01:52 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.092209987s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.185378663s trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:01:09 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 1.185404611s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 958.224282ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 15:00:07 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 958.249908ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 862.549955ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:59:21 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 862.586032ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 921.313851ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:58:23 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 921.337667ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 951.420616ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:58:07 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 951.445562ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 858.987708ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:57:40 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 859.012936ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 870.477282ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:56:53 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 870.507654ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 841.842364ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:55:50 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 841.869826ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 927.721912ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:54:52 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 927.74726ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 925.576467ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:53:45 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 925.605692ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 937.165084ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:52:40 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 937.192236ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 922.928466ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:51:39 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 922.958251ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 846.466229ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:51:17 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 846.572442ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 805.178174ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:50:40 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 805.204236ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 801.693654ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:50:18 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 801.71265ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 813.514672ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:49:26 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 813.542057ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 952.377723ms trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:48:52 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 952.421672ms core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000962956s trace: https://jaeger.dev.storj.io/trace/dc3aa5c41895632 FAIL (2025-12-14 14:48:26 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 30.000985281s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 510.360111ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:20:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 510.388196ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 388.748071ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:20:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 388.779257ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 423.772315ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:19:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 423.804626ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 390.00864ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:18:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 390.043362ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 397.120313ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:18:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 397.14737ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 401.153573ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:17:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 401.182335ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 539.151888ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:16:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 539.203947ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 431.567266ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:16:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 431.605288ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 423.684375ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:15:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 423.710964ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 537.600521ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:14:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 537.631544ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 483.078848ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:13:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 483.173738ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 437.267517ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:13:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 437.289441ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)