Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...............................................XX...........] duration:
887.680048ms trace:
https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66
FAIL (2026-04-07 22:57:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 887.720001ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
455.507475ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:57:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 455.539218ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
829.369545ms trace:
https://jaeger.dev.storj.io/trace/105ed3c1d703150f
FAIL (2026-04-07 22:56:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 829.420742ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [................................................XX..........] duration:
531.345848ms trace:
https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66
FAIL (2026-04-07 22:56:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 531.375607ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
462.46699ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:56:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 462.57609ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.................................................XX.........] duration:
510.29056ms trace:
https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66
FAIL (2026-04-07 22:56:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 510.32738ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
621.610389ms trace:
https://jaeger.dev.storj.io/trace/105ed3c1d703150f
FAIL (2026-04-07 22:55:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 621.639258ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
461.661413ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:55:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 461.833142ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..................................................XX........] duration:
648.897729ms trace:
https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66
FAIL (2026-04-07 22:55:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 649.025ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
404.269217ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:54:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 404.301504ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...................................................XX.......] duration:
461.753843ms trace:
https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66
FAIL (2026-04-07 22:54:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 461.816417ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
584.244962ms trace:
https://jaeger.dev.storj.io/trace/105ed3c1d703150f
FAIL (2026-04-07 22:54:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 584.275027ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
393.767193ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:54:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 393.876288ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
601.377391ms trace:
https://jaeger.dev.storj.io/trace/105ed3c1d703150f
FAIL (2026-04-07 22:54:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 601.403208ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....................................................XX......] duration:
553.419852ms trace:
https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66
FAIL (2026-04-07 22:53:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 553.440496ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
585.69732ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:53:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 585.722749ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
559.041392ms trace:
https://jaeger.dev.storj.io/trace/105ed3c1d703150f
FAIL (2026-04-07 22:53:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 559.080965ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
530.787079ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:52:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 530.814757ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....................................................XX.....] duration:
498.358619ms trace:
https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66
FAIL (2026-04-07 22:52:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 498.387444ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
576.236258ms trace:
https://jaeger.dev.storj.io/trace/105ed3c1d703150f
FAIL (2026-04-07 22:52:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 576.26594ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
512.052988ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:52:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 512.082479ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......................................................XX....] duration:
481.084444ms trace:
https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66
FAIL (2026-04-07 22:52:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 481.135146ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
747.839598ms trace:
https://jaeger.dev.storj.io/trace/105ed3c1d703150f
FAIL (2026-04-07 22:51:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 747.890277ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
664.30903ms trace:
https://jaeger.dev.storj.io/trace/105ed3c1d703150f
FAIL (2026-04-07 22:51:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 664.348877ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................................................XX...] duration:
481.937932ms trace:
https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66
FAIL (2026-04-07 22:51:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 481.975921ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
479.891194ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:50:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 479.920366ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
613.092123ms trace:
https://jaeger.dev.storj.io/trace/105ed3c1d703150f
FAIL (2026-04-07 22:50:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 613.127253ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................................................XX..] duration:
513.248294ms trace:
https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66
FAIL (2026-04-07 22:50:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 513.282339ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
457.827743ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:49:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 457.855544ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
616.620021ms trace:
https://jaeger.dev.storj.io/trace/105ed3c1d703150f
FAIL (2026-04-07 22:49:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 616.656618ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
497.042495ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:49:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 497.068787ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
420.843273ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:48:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 420.956701ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................................................XX.] duration:
698.485116ms trace:
https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66
FAIL (2026-04-07 22:48:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 698.512023ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
597.362254ms trace:
https://jaeger.dev.storj.io/trace/105ed3c1d703150f
FAIL (2026-04-07 22:48:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 597.42195ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
416.317135ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:48:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 416.344576ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
555.957976ms trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:48:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 555.987047ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
677.166334ms trace:
https://jaeger.dev.storj.io/trace/105ed3c1d703150f
FAIL (2026-04-07 22:48:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 677.18594ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........................................................XX] duration:
5.002630545s trace:
https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66
FAIL (2026-04-07 22:47:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002655447s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.022196649s trace:
https://jaeger.dev.storj.io/trace/345ead65e237bb8a
FAIL (2026-04-07 22:47:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload
Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload
Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments...
Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false
Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment...
Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment.
Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece...
Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment
Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments...
Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil>
Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments...
Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.02224089s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X] duration:
5.001018539s trace:
https://jaeger.dev.storj.io/trace/6166a6675ad280ec
FAIL (2026-04-07 22:47:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001039624s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
590.042275ms trace:
https://jaeger.dev.storj.io/trace/4777dea64937dc1a
FAIL (2026-04-06 16:26:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 590.082469ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
621.270533ms trace:
https://jaeger.dev.storj.io/trace/28874b4e710a2bcf
FAIL (2026-04-06 16:25:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 621.302019ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
609.246084ms trace:
https://jaeger.dev.storj.io/trace/4777dea64937dc1a
FAIL (2026-04-06 16:25:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 609.284432ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
593.359183ms trace:
https://jaeger.dev.storj.io/trace/4777dea64937dc1a
FAIL (2026-04-06 16:25:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 593.385414ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [XX..........................................................] duration:
480.525888ms trace:
https://jaeger.dev.storj.io/trace/28874b4e710a2bcf
FAIL (2026-04-06 16:24:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 480.861876ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
1.255037447s trace:
https://jaeger.dev.storj.io/trace/449433b6185f502e
FAIL (2026-04-06 16:24:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload
Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload
Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments...
Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false
Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment...
Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment.
Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment
Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments...
Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded
Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments...
Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.255063585s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
650.960911ms trace:
https://jaeger.dev.storj.io/trace/4777dea64937dc1a
FAIL (2026-04-06 16:24:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 650.995407ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
1.394837476s trace:
https://jaeger.dev.storj.io/trace/449433b6185f502e
FAIL (2026-04-06 16:24:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload
Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload
Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments...
Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false
Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment...
Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment.
Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment
Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments...
Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded
Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments...
Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.394871433s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
1.475759407s trace:
https://jaeger.dev.storj.io/trace/449433b6185f502e
FAIL (2026-04-06 16:23:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload
Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload
Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments...
Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false
Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment...
Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment.
Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece...
Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment
Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments...
Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil>
Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded
Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments...
Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.475793197s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [XXX.........................................................] duration:
511.939349ms trace:
https://jaeger.dev.storj.io/trace/28874b4e710a2bcf
FAIL (2026-04-06 16:23:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 512.040588ms
core
uplink
us-select-1
us1-placement-20
|