Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
4.219726016s trace:
https://jaeger.dev.storj.io/trace/4e230bf55c2129c5
FAIL (2025-11-23 13:06:41 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.21975859s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
4.360961772s trace:
https://jaeger.dev.storj.io/trace/4e230bf55c2129c5
FAIL (2025-11-23 13:06:15 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.360990287s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
4.276346234s trace:
https://jaeger.dev.storj.io/trace/4e230bf55c2129c5
FAIL (2025-11-23 13:05:08 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.276374216s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
4.449386208s trace:
https://jaeger.dev.storj.io/trace/4e230bf55c2129c5
FAIL (2025-11-23 13:04:05 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.449422597s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
4.384571482s trace:
https://jaeger.dev.storj.io/trace/4e230bf55c2129c5
FAIL (2025-11-23 13:03:16 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.384597402s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
4.245719473s trace:
https://jaeger.dev.storj.io/trace/4e230bf55c2129c5
FAIL (2025-11-23 13:02:31 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.245748106s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
4.230660547s trace:
https://jaeger.dev.storj.io/trace/4e230bf55c2129c5
FAIL (2025-11-23 13:01:44 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.230684046s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
4.31579062s trace:
https://jaeger.dev.storj.io/trace/4e230bf55c2129c5
FAIL (2025-11-23 13:01:18 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.315826157s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
4.695829648s trace:
https://jaeger.dev.storj.io/trace/4e230bf55c2129c5
FAIL (2025-11-23 13:00:57 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.695856656s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
4.311154693s trace:
https://jaeger.dev.storj.io/trace/4e230bf55c2129c5
FAIL (2025-11-23 12:59:46 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.311179586s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
4.299095054s trace:
https://jaeger.dev.storj.io/trace/4e230bf55c2129c5
FAIL (2025-11-23 12:58:36 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.299123187s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
33.953858498s trace:
https://jaeger.dev.storj.io/trace/4e230bf55c2129c5
FAIL (2025-11-23 12:57:54 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 33.953880725s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
428.727689ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 13:22:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 428.74553ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
422.85165ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 13:21:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 422.8779ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [X.X.........................................................] duration:
445.439234ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 13:20:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 445.451192ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [XX.X........................................................] duration:
435.439596ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 13:19:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 435.466078ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
524.637126ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 13:19:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 524.655238ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.XX.X.......................................................] duration:
620.009938ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 13:18:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 620.034259ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
537.600099ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 13:18:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 537.627432ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [..XX.X......................................................] duration:
422.214015ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 13:18:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 422.272445ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
593.254782ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 13:17:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 593.284245ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [...XX.X.....................................................] duration:
410.415324ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 13:17:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 410.443478ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
485.604786ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 13:16:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 485.628768ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [X..X........................................................] duration:
564.301537ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 13:16:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 564.323779ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [X...XX.X....................................................] duration:
591.866509ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 13:16:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 591.919392ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.X...XX.X...................................................] duration:
477.804787ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 13:16:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 477.832881ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
454.814865ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 13:16:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 454.906807ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
502.021218ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 13:16:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 502.09421ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..X...XX.X..................................................] duration:
452.402695ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 13:15:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 452.441111ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration:
532.141002ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 13:15:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 532.19061ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X.X.........................................................] duration:
416.719039ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 13:15:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 416.738848ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
470.448875ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 13:15:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 470.478655ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...X...XX.X.................................................] duration:
415.823842ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 13:14:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 415.848259ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
423.847066ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 13:14:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 423.873386ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..X..X......................................................] duration:
503.296655ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 13:14:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 503.323924ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X.X........................................................] duration:
501.273777ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 13:14:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 501.323579ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
612.352877ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 13:14:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 612.384124ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X.X.......................................................] duration:
453.001218ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 13:14:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 453.020088ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
363.23192ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 13:14:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 363.261933ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
365.164994ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 13:13:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 365.189101ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....X...XX.X................................................] duration:
429.698734ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 13:13:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 429.759554ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...X..X.....................................................] duration:
531.059424ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 13:13:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 531.089731ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X..X.X......................................................] duration:
447.53912ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 13:13:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 447.567984ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
362.205622ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 13:13:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 362.230587ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....X..X....................................................] duration:
498.223327ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 13:13:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 498.247326ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [X.X.........................................................] duration:
431.941825ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 13:13:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 432.06362ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.....X...XX.X...............................................] duration:
427.481705ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 13:13:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 427.50718ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
367.234706ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 13:12:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 367.267096ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [X.....X.....................................................] duration:
397.83904ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 13:12:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 397.86315ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [XX..X.X.....................................................] duration:
466.615708ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 13:12:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 466.713492ms
edge
gateway
us-select-1
|