Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 664.989246ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:19:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 665.070748ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 953.812028ms trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:19:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 953.855948ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 452.888803ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:18:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 452.933141ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 783.991361ms trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:18:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 784.030431ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.094890218s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:18:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.094925005s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.300838706s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:17:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.300870612s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 744.834134ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:17:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 744.865839ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.273473066s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:17:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.273505826s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.286706658s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:17:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.286739054s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.162951861s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:16:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.163028059s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 708.599215ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:16:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 708.632853ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.268042064s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:16:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.268095508s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.070470184s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:15:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.070497483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 730.338266ms trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:15:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 730.371621ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 648.889697ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:15:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 648.923129ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.373637296s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:15:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.373674517s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 496.661132ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:15:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 496.694109ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.314982371s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:15:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.315013967s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.212703992s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:14:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.212750791s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 481.667063ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:14:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 481.694825ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.357563407s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:14:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.357610792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 899.966773ms trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:14:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 900.025146ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 674.498028ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:13:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 674.531292ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 948.456831ms trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:13:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 948.488274ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 779.846522ms trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:13:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 779.874553ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 927.983633ms trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:12:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 928.011742ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 663.574903ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:12:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 663.624937ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.320207514s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:12:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.32024631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 560.532535ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:12:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 560.558593ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.302800399s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:12:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.302837347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.169164671s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:12:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.169193318s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 515.593842ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:11:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 515.629796ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.274438989s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:11:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.27447409s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 547.621299ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:10:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 547.653698ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.282826082s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:10:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.282878141s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 987.628268ms trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:10:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 988.004201ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 604.903609ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:10:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 604.96452ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 983.541252ms trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:10:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 983.57852ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.153904855s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:10:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.154029237s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 626.859455ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:09:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 626.888965ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.213610792s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:09:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.213641376s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.334337162s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:09:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.334369596s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 564.693081ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:09:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 564.746321ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 735.681074ms trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:09:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 735.805843ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.201851989s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:08:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.201887406s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 676.08901ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 16:08:41 UTC) orbiter warning: unexpected EOF Duration: 676.11275ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 503.164389ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:08:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 503.254895ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.097835673s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:08:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.097966109s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 644.75452ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 16:08:16 UTC) orbiter warning: unexpected EOF Duration: 645.314658ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 822.300287ms trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:08:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 822.328013ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)