Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
713.482558ms target:
https://ap1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/c9cd61a7d6477ab
FAIL (2026-03-16 23:53:28 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.146.6:443: i/o timeout
Duration: 713.512728ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
1.780835051s trace:
https://jaeger.dev.storj.io/trace/6f888b1e7fd43d
FAIL (2026-03-16 23:53:26 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 1.780871327s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration:
593.609622ms trace:
https://jaeger.dev.storj.io/trace/22ca2a71d8e0411f
FAIL (2026-03-16 23:53:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 593.665001ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
3 out of 60 is failed (threshold=3) [XXX.........................................................] duration:
4.503408189s trace:
https://jaeger.dev.storj.io/trace/488c16a528e2a255
FAIL (2026-03-16 23:53:04 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.503436291s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
256.549793ms trace:
https://jaeger.dev.storj.io/trace/5ee26ff87c6300e4
FAIL (2026-03-16 23:53:00 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 256.565074ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.XXXXX......................................................] duration:
605.587024ms trace:
https://jaeger.dev.storj.io/trace/22ca2a71d8e0411f
FAIL (2026-03-16 23:52:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 605.61796ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [XXXXXX......................................................] duration:
460.179189ms trace:
https://jaeger.dev.storj.io/trace/70403794f26a4156
FAIL (2026-03-16 23:52:51 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 460.264879ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
219.85505ms trace:
https://jaeger.dev.storj.io/trace/18df21c1c2c0ed9a
FAIL (2026-03-16 23:52:44 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 219.891404ms
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
1.164244142s trace:
https://jaeger.dev.storj.io/trace/6fdbf3c7687a0ce5
FAIL (2026-03-16 23:52:41 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.164272135s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.XXX........................................................] duration:
4.510723576s trace:
https://jaeger.dev.storj.io/trace/488c16a528e2a255
FAIL (2026-03-16 23:52:41 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.51075876s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
717.827726ms target:
https://ap1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/c9cd61a7d6477ab
FAIL (2026-03-16 23:52:41 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.146.6:443: i/o timeout
Duration: 717.858814ms
core
satellite
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..XXXXX.....................................................] duration:
605.562783ms trace:
https://jaeger.dev.storj.io/trace/22ca2a71d8e0411f
FAIL (2026-03-16 23:52:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 605.598548ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [XXXXXXX.....................................................] duration:
689.354513ms trace:
https://jaeger.dev.storj.io/trace/70403794f26a4156
FAIL (2026-03-16 23:52:33 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 689.381803ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
2.01124101s trace:
https://jaeger.dev.storj.io/trace/1f4071450dd807fc
FAIL (2026-03-16 23:52:28 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 23:43:18.026386127 (252h43m15.623157157s) [upload=283106]: Starting upload Mar 16 23:43:18.026569032 (252h43m15.623340062s) [upload=283106]: Done starting upload Mar 16 23:43:18.026683049 (252h43m15.623454058s) [upload=283106]: Uploading segments... Mar 16 23:43:18.027247579 (252h43m15.62401861s) [upload=283106]: Got next segment. Inline: false Mar 16 23:43:18.027369023 (252h43m15.624140057s) [upload=283106]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 23:43:18.027377197 (252h43m15.624148225s) [upload=283106]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 23:43:18.129722287 (252h43m15.726493311s) [upload=283106]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 23:43:18.129747413 (252h43m15.726518446s) [upload=283106]: Done uploading segments... Mar 16 23:43:18.129809790 (252h43m15.726580818s) [upload=283106]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 2.01126177s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
1.782306385s trace:
https://jaeger.dev.storj.io/trace/6f888b1e7fd43d
FAIL (2026-03-16 23:52:27 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 1.78233551s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
1.297413685s trace:
https://jaeger.dev.storj.io/trace/71a8d4082dd7d62d
FAIL (2026-03-16 23:52:27 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.297433739s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
724.227278ms target:
https://ap1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/c9cd61a7d6477ab
FAIL (2026-03-16 23:52:13 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.146.6:443: i/o timeout
Duration: 724.253392ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
1.900081704s trace:
https://jaeger.dev.storj.io/trace/1f4071450dd807fc
FAIL (2026-03-16 23:52:11 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 23:43:18.026386127 (252h43m15.623157157s) [upload=283106]: Starting upload Mar 16 23:43:18.026569032 (252h43m15.623340062s) [upload=283106]: Done starting upload Mar 16 23:43:18.026683049 (252h43m15.623454058s) [upload=283106]: Uploading segments... Mar 16 23:43:18.027247579 (252h43m15.62401861s) [upload=283106]: Got next segment. Inline: false Mar 16 23:43:18.027369023 (252h43m15.624140057s) [upload=283106]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 23:43:18.027377197 (252h43m15.624148225s) [upload=283106]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 23:43:18.129722287 (252h43m15.726493311s) [upload=283106]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 23:43:18.129747413 (252h43m15.726518446s) [upload=283106]: Done uploading segments... Mar 16 23:43:18.129809790 (252h43m15.726580818s) [upload=283106]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.900112037s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
1.708310047s trace:
https://jaeger.dev.storj.io/trace/6f888b1e7fd43d
FAIL (2026-03-16 23:52:06 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 1.708358101s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
300.080337ms trace:
https://jaeger.dev.storj.io/trace/5ee26ff87c6300e4
FAIL (2026-03-16 23:51:57 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 300.108355ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
1.896270116s trace:
https://jaeger.dev.storj.io/trace/1f4071450dd807fc
FAIL (2026-03-16 23:51:53 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 23:43:18.026386127 (252h43m15.623157157s) [upload=283106]: Starting upload Mar 16 23:43:18.026569032 (252h43m15.623340062s) [upload=283106]: Done starting upload Mar 16 23:43:18.026683049 (252h43m15.623454058s) [upload=283106]: Uploading segments... Mar 16 23:43:18.027247579 (252h43m15.62401861s) [upload=283106]: Got next segment. Inline: false Mar 16 23:43:18.027369023 (252h43m15.624140057s) [upload=283106]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 23:43:18.027377197 (252h43m15.624148225s) [upload=283106]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 23:43:18.129722287 (252h43m15.726493311s) [upload=283106]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 23:43:18.129747413 (252h43m15.726518446s) [upload=283106]: Done uploading segments... Mar 16 23:43:18.129809790 (252h43m15.726580818s) [upload=283106]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.896301346s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [XXXXXXXX....................................................] duration:
489.930753ms trace:
https://jaeger.dev.storj.io/trace/70403794f26a4156
FAIL (2026-03-16 23:51:53 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 489.94824ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
1.22218206s trace:
https://jaeger.dev.storj.io/trace/6fdbf3c7687a0ce5
FAIL (2026-03-16 23:51:51 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.222214432s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
218.972576ms trace:
https://jaeger.dev.storj.io/trace/18df21c1c2c0ed9a
FAIL (2026-03-16 23:51:51 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 218.988224ms
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
1.689680491s trace:
https://jaeger.dev.storj.io/trace/6f888b1e7fd43d
FAIL (2026-03-16 23:51:46 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 1.689707682s
core
uplink
us1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration:
4.254624127s trace:
https://jaeger.dev.storj.io/trace/488c16a528e2a255
FAIL (2026-03-16 23:51:41 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.254650921s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...XXXXX....................................................] duration:
594.574796ms trace:
https://jaeger.dev.storj.io/trace/22ca2a71d8e0411f
FAIL (2026-03-16 23:51:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 594.632343ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
1.928856836s trace:
https://jaeger.dev.storj.io/trace/1f4071450dd807fc
FAIL (2026-03-16 23:51:32 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 23:43:18.026386127 (252h43m15.623157157s) [upload=283106]: Starting upload Mar 16 23:43:18.026569032 (252h43m15.623340062s) [upload=283106]: Done starting upload Mar 16 23:43:18.026683049 (252h43m15.623454058s) [upload=283106]: Uploading segments... Mar 16 23:43:18.027247579 (252h43m15.62401861s) [upload=283106]: Got next segment. Inline: false Mar 16 23:43:18.027369023 (252h43m15.624140057s) [upload=283106]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 23:43:18.027377197 (252h43m15.624148225s) [upload=283106]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 23:43:18.129722287 (252h43m15.726493311s) [upload=283106]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 23:43:18.129747413 (252h43m15.726518446s) [upload=283106]: Done uploading segments... Mar 16 23:43:18.129809790 (252h43m15.726580818s) [upload=283106]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.928886319s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
255.71424ms trace:
https://jaeger.dev.storj.io/trace/7d774feb475c25d5
FAIL (2026-03-16 23:51:22 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 255.801627ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
1.201544575s trace:
https://jaeger.dev.storj.io/trace/71a8d4082dd7d62d
FAIL (2026-03-16 23:51:18 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.201574262s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [..XX........................................................] duration:
719.654214ms target:
https://ap1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/c9cd61a7d6477ab
FAIL (2026-03-16 23:51:16 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.146.6:443: i/o timeout
Duration: 719.692498ms
core
satellite
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [....XXXXX...................................................] duration:
585.089848ms trace:
https://jaeger.dev.storj.io/trace/22ca2a71d8e0411f
FAIL (2026-03-16 23:51:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 585.13274ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
279.886525ms trace:
https://jaeger.dev.storj.io/trace/18df21c1c2c0ed9a
FAIL (2026-03-16 23:51:05 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 279.921772ms
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
216.265659ms trace:
https://jaeger.dev.storj.io/trace/7d774feb475c25d5
FAIL (2026-03-16 23:50:51 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 216.290654ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
264.772517ms trace:
https://jaeger.dev.storj.io/trace/5ee26ff87c6300e4
FAIL (2026-03-16 23:50:47 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 264.804765ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
889.829779ms trace:
https://jaeger.dev.storj.io/trace/6fdbf3c7687a0ce5
FAIL (2026-03-16 23:50:42 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 889.856499ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
2.007969259s trace:
https://jaeger.dev.storj.io/trace/6f888b1e7fd43d
FAIL (2026-03-16 23:50:41 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.008055987s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [.XXXXXXXX...................................................] duration:
502.121421ms trace:
https://jaeger.dev.storj.io/trace/70403794f26a4156
FAIL (2026-03-16 23:50:41 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 502.146362ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.....XXXXX..................................................] duration:
547.474493ms trace:
https://jaeger.dev.storj.io/trace/22ca2a71d8e0411f
FAIL (2026-03-16 23:50:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 547.515809ms
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://ap1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.ap1.satellite_login)antiblip:
2 out of 60 is failed (threshold=3) [...XX.......................................................] duration:
719.483593ms target:
https://ap1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/c9cd61a7d6477ab
FAIL (2026-03-16 23:50:32 UTC)
orbiter warning: Get "https://ap1.storj.io/login": dial tcp 34.2.146.6:443: i/o timeout
Duration: 719.511321ms
core
satellite
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...XXX......................................................] duration:
4.596564949s trace:
https://jaeger.dev.storj.io/trace/488c16a528e2a255
FAIL (2026-03-16 23:50:31 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.596600183s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
1.057335528s trace:
https://jaeger.dev.storj.io/trace/71a8d4082dd7d62d
FAIL (2026-03-16 23:50:28 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.057441359s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
1.994370404s trace:
https://jaeger.dev.storj.io/trace/1f4071450dd807fc
FAIL (2026-03-16 23:50:26 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 ======LOGS====== Mar 16 23:43:18.026386127 (252h43m15.623157157s) [upload=283106]: Starting upload Mar 16 23:43:18.026569032 (252h43m15.623340062s) [upload=283106]: Done starting upload Mar 16 23:43:18.026683049 (252h43m15.623454058s) [upload=283106]: Uploading segments... Mar 16 23:43:18.027247579 (252h43m15.62401861s) [upload=283106]: Got next segment. Inline: false Mar 16 23:43:18.027369023 (252h43m15.624140057s) [upload=283106]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 16 23:43:18.027377197 (252h43m15.624148225s) [upload=283106]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Mar 16 23:43:18.129722287 (252h43m15.726493311s) [upload=283106]: Next segment err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0 Mar 16 23:43:18.129747413 (252h43m15.726518446s) [upload=283106]: Done uploading segments... Mar 16 23:43:18.129809790 (252h43m15.726580818s) [upload=283106]: Upload finished. err: metaclient: not enough nodes: not enough nodes: requested from cache 20, found 0
Duration: 1.994396794s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.01385637s trace:
https://jaeger.dev.storj.io/trace/18e92533cb874ee7
FAIL (2026-03-16 23:50:25 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.013886048s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration:
4.384271396s trace:
https://jaeger.dev.storj.io/trace/488c16a528e2a255
FAIL (2026-03-16 23:50:11 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.384316139s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [......XXXXX.................................................] duration:
751.666724ms trace:
https://jaeger.dev.storj.io/trace/22ca2a71d8e0411f
FAIL (2026-03-16 23:50:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 751.747819ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
216.055746ms trace:
https://jaeger.dev.storj.io/trace/7d774feb475c25d5
FAIL (2026-03-16 23:50:08 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 216.086718ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [..XXXXXXXX..................................................] duration:
752.745072ms trace:
https://jaeger.dev.storj.io/trace/70403794f26a4156
FAIL (2026-03-16 23:50:07 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 752.772417ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
245.369428ms trace:
https://jaeger.dev.storj.io/trace/18df21c1c2c0ed9a
FAIL (2026-03-16 23:49:56 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 245.395155ms
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
251.163395ms trace:
https://jaeger.dev.storj.io/trace/5ee26ff87c6300e4
FAIL (2026-03-16 23:49:53 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 251.203739ms
core
uplink
us-select-1
|