Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.......................................................XXXXX] duration:
5.000847688s trace:
https://jaeger.dev.storj.io/trace/396ac7c1cb75d0f8
FAIL (2026-03-16 23:10:10 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.000874647s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration:
4.32275768s trace:
https://jaeger.dev.storj.io/trace/488c16a528e2a255
FAIL (2026-03-16 23:09:45 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.322786744s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [........................................................XX..] duration:
719.152295ms target:
https://ap1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/c9cd61a7d6477ab
FAIL (2026-03-16 23:09:44 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.146.6:443: i/o timeout
Duration: 719.181612ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
257.093311ms trace:
https://jaeger.dev.storj.io/trace/7d774feb475c25d5
FAIL (2026-03-16 23:09:36 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 257.118129ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
1m0.00155942s trace:
https://jaeger.dev.storj.io/trace/7ec0589b87da2eb3
FAIL (2026-03-16 23:09:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001590387s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration:
2.042051953s trace:
https://jaeger.dev.storj.io/trace/18e92533cb874ee7
FAIL (2026-03-16 23:09:06 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.042132046s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [........................................................XXXX] duration:
5.000994949s trace:
https://jaeger.dev.storj.io/trace/4d20648e7c6b8c00
FAIL (2026-03-16 23:09:04 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.0010829s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................XX.] duration:
724.284795ms target:
https://ap1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/c9cd61a7d6477ab
FAIL (2026-03-16 23:09:02 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.146.6:443: i/o timeout
Duration: 724.314275ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [........................................................XXX.] duration:
26.025787813s trace:
https://jaeger.dev.storj.io/trace/18e92533cb874ee7
FAIL (2026-03-16 23:08:55 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 26.025882748s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................................................XXX.] duration:
4.288613869s trace:
https://jaeger.dev.storj.io/trace/488c16a528e2a255
FAIL (2026-03-16 23:08:54 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.288633769s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
30.001962805s trace:
https://jaeger.dev.storj.io/trace/488c16a528e2a255
FAIL (2026-03-16 23:08:49 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.001989492s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
304.621227ms trace:
https://jaeger.dev.storj.io/trace/7d774feb475c25d5
FAIL (2026-03-16 23:08:41 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 304.646702ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.........................................................XXX] duration:
5.00107934s trace:
https://jaeger.dev.storj.io/trace/4ddf7af09336ebc1
FAIL (2026-03-16 23:08:37 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.001103447s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.001416482s target:
https://ap1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/c9cd61a7d6477ab
FAIL (2026-03-16 23:08:31 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.146.6:443: i/o timeout
Duration: 30.001443518s
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
30.000909263s trace:
https://jaeger.dev.storj.io/trace/18e92533cb874ee7
FAIL (2026-03-16 23:08:29 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.000930437s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.001354586s trace:
https://jaeger.dev.storj.io/trace/4e68c4a25ca68cd6
FAIL (2026-03-16 23:08:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001446551s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
240.252826ms trace:
https://jaeger.dev.storj.io/trace/7d774feb475c25d5
FAIL (2026-03-16 23:08:17 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 240.278651ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.001195126s trace:
https://jaeger.dev.storj.io/trace/7d774feb475c25d5
FAIL (2026-03-16 23:08:16 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1m0.001217804s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.001971999s trace:
https://jaeger.dev.storj.io/trace/2d8c87b3eff33778
FAIL (2026-03-16 23:07:45 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.00198622s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.001496721s trace:
https://jaeger.dev.storj.io/trace/43770f7288a5c0ad
FAIL (2026-03-16 23:07:43 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.001512749s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000867236s target:
https://ap1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7661b76d80b37c76
FAIL (2026-03-16 23:07:42 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.146.6:443: i/o timeout
Duration: 30.000894006s
core
satellite
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........................................................XX] duration:
5.000352413s trace:
https://jaeger.dev.storj.io/trace/72775ffd91f977e6
FAIL (2026-03-16 23:07:32 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.000365491s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001161325s trace:
https://jaeger.dev.storj.io/trace/7f122a5f57cd00e6
FAIL (2026-03-16 23:07:15 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.001187094s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.002077233s trace:
https://jaeger.dev.storj.io/trace/69d160e603d304c9
FAIL (2026-03-16 23:07:13 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.002103088s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X] duration:
5.000531461s trace:
https://jaeger.dev.storj.io/trace/d4b9618c680356a
FAIL (2026-03-16 23:06:41 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.000622793s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
1.881627816s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:29:34 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.881720893s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
1.913370602s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:28:55 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.913456941s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
1.855014479s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:27:55 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.855043005s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
1.895324874s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:27:29 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.895354846s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
1.945123441s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:26:14 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.945162561s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
1.895320416s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:25:44 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.895349019s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
1.873681374s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:24:35 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.87370899s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
1.875159447s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:23:29 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.875229181s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
1.918537093s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:23:03 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.918565607s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
2.142901438s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:22:11 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 2.142928083s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
1.976282331s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:21:20 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.976349504s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
1.929207411s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:20:16 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.929234409s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
1.873354927s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:19:30 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.873382244s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
1.875140951s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:19:03 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.875291281s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
1.885770724s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:18:38 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.885810266s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
1.950492518s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:17:24 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.950540346s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
1.889031746s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:16:46 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.889147449s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
1.860200317s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:16:18 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.860259903s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
1.900849315s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:15:10 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.900880677s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
1.91891808s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:14:42 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.918946379s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
2.176411979s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:13:45 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 2.176456662s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
2.132328182s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:12:47 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 2.132358529s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
1.881843539s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:12:04 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.881873915s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
1.917211171s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:10:58 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.917235559s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
2.087894472s trace:
https://jaeger.dev.storj.io/trace/c9c88723b64f139
FAIL (2026-03-16 22:09:54 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 21:42:53.346802918 (250h42m50.943573942s) [upload=280850]: Starting upload Mar 16 21:42:53.346875441 (250h42m50.943646458s) [upload=280850]: Done starting upload Mar 16 21:42:53.346961558 (250h42m50.943732573s) [upload=280850]: Uploading segments... Mar 16 21:42:53.347318481 (250h42m50.944089504s) [upload=280850]: Got next segment. Inline: false Mar 16 21:42:53.347364020 (250h42m50.944135046s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 21:42:53.347369097 (250h42m50.944140121s) [upload=280850]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 21:42:53.455077771 (250h42m51.051848794s) [upload=280850]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 21:42:53.455097746 (250h42m51.051868771s) [upload=280850]: Done uploading segments... Mar 16 21:42:53.455139311 (250h42m51.051910325s) [upload=280850]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 2.087926744s
core
uplink
uk-select-1
uk1-placement-17
|