Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................................XX..................] duration:
496.186605ms trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:27:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 496.220959ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................................XX.................] duration:
448.168718ms trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:26:28 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 448.238062ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........................................XX................] duration:
1.083950044s trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:26:14 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.083994273s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...........................................XX...............] duration:
887.742532ms trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:25:12 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 887.785739ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [............................................XX..............] duration:
828.010189ms trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:24:32 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 828.029072ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.............................................XX.............] duration:
703.67424ms trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:23:49 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 703.701122ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..............................................XX............] duration:
829.502639ms trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:23:19 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 829.529623ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...............................................XX...........] duration:
832.441024ms trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:22:49 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 832.541777ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [................................................XX..........] duration:
980.148759ms trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:21:39 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 980.196101ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.................................................XX.........] duration:
561.623103ms trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:20:41 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 561.64884ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..................................................XX........] duration:
1.580518456s trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:19:36 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.580550088s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...................................................XX.......] duration:
769.799161ms trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:19:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 769.905753ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....................................................XX......] duration:
1.556049129s trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:18:51 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.556166468s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....................................................XX.....] duration:
980.93682ms trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:17:46 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 980.967201ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......................................................XX....] duration:
1.528723497s trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:17:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.528750253s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................................................XX...] duration:
1.587500225s trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:16:18 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.587535971s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................................................XX..] duration:
3.016248155s trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:15:48 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.016284439s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................................................XX.] duration:
2.124927571s trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:15:26 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.124953864s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........................................................XX] duration:
5.000981712s trace:
https://jaeger.dev.storj.io/trace/215a41ee5a6f5888
FAIL (2026-04-03 22:15:08 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001025063s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X] duration:
5.00328762s trace:
https://jaeger.dev.storj.io/trace/6eb5146b1d29f566
FAIL (2026-04-03 22:14:13 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00332135s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
860.008847ms trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:40:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 860.039552ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
493.099549ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:39:58 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 493.497547ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
1.296404102s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:39:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.296469723s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
1.199168935s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:39:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.199199397s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
1.286658559s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:39:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.286691349s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
577.588832ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:38:57 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 577.616855ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
1.206348093s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:38:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.206379359s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
939.825374ms trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:38:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 939.879389ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
1.29582655s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:38:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.295931728s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..X.........................................................] duration:
655.780337ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:37:50 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 655.803248ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
1.233126239s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:37:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.233157369s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
493.854575ms trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:37:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 493.887198ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
1.160920509s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:37:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.160952394s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...X........................................................] duration:
551.641262ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:37:03 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 551.671786ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....X.......................................................] duration:
473.257793ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:36:41 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 473.298193ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
1.147246022s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:36:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.147277273s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....X......................................................] duration:
574.511617ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:36:24 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 574.634958ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
907.879841ms trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:36:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 907.91283ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
743.33233ms trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:35:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 743.3906ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......X.....................................................] duration:
531.813749ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:35:45 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 531.845039ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
752.383477ms trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:35:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 752.406959ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......X....................................................] duration:
520.293325ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:35:19 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 520.327041ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
1.163605803s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:35:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.163694888s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
1.196893067s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:34:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.19692646s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
1.125356901s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:34:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.125459362s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........X...................................................] duration:
1.724402957s trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:34:09 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.724433093s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
908.258661ms trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:34:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 908.34992ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
1.33697778s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:33:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.337042843s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........X..................................................] duration:
1.667963877s trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:33:38 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.668003643s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
1.135469883s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:33:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.135574205s
core
uplink
us-select-1
us1-placement-20
|