Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......XX....................................................] duration:
489.341762ms trace:
https://jaeger.dev.storj.io/trace/732e062283542b6f
FAIL (2026-02-17 23:47:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 489.369351ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
20 out of 60 is failed (threshold=3) [.........X..XX.XX.X...XX.X..XXXXX.XX.X.X..XX................] duration:
1.035502562s trace:
https://jaeger.dev.storj.io/trace/2be28be9b880734a
FAIL (2026-02-17 23:47:02 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Feb 17 23:34:23.059166009 (3h18m23.862525184s) [upload=3409]: Starting upload
Feb 17 23:34:23.059410193 (3h18m23.862769371s) [upload=3409]: Done starting upload
Feb 17 23:34:23.059535700 (3h18m23.86289487s) [upload=3409]: Uploading segments...
Feb 17 23:34:23.060262272 (3h18m23.863621456s) [upload=3409]: Got next segment. Inline: false
Feb 17 23:34:23.060415851 (3h18m23.86377504s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:34:23.060424507 (3h18m23.863783691s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:34:23.091662596 (3h18m23.89502178s) [seg_pos={0 0}, upload=3409]: Begin upload segment...
Feb 17 23:34:23.091736818 (3h18m23.895096053s) [seg_pos={0 0}, upload=3409]: Done begin upload segment.
Feb 17 23:34:23.091776467 (3h18m23.895135652s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.091923016 (3h18m23.8952822s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.092846932 (3h18m23.896206109s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093631811 (3h18m23.896990994s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093929468 (3h18m23.897288656s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.094936563 (3h18m23.898295747s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.095529634 (3h18m23.898888818s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096283238 (3h18m23.899642414s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096313780 (3h18m23.899672964s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096989058 (3h18m23.900348238s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097111179 (3h18m23.900470354s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097545617 (3h18m23.90090479s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097826531 (3h18m23.90118571s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098091522 (3h18m23.901450698s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098582559 (3h18m23.901941739s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098625393 (3h18m23.901984578s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099438197 (3h18m23.902797382s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099473251 (3h18m23.902832435s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.168134648 (3h18m23.971493869s) [upload=3409]: Next returned nil segment
Feb 17 23:34:23.168173410 (3h18m23.97153259s) [upload=3409]: Waiting for error group managing segments...
Feb 17 23:34:23.192662134 (3h18m23.996021325s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.199821483 (3h18m24.003180644s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.216892202 (3h18m24.020251384s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.315136137 (3h18m24.118495316s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.369180508 (3h18m24.17253969s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.676435083 (3h18m24.479794257s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.686494858 (3h18m24.48985402s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690268874 (3h18m24.493628038s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690325658 (3h18m24.493684832s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.702724806 (3h18m24.506083983s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.704052843 (3h18m24.507412028s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:24.098092821 (3h18m24.901451996s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:35:23.025077135 (3h19m23.82843643s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: rpc: dial tcp 77.90.31.15:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025124657 (3h19m23.828483841s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025424430 (3h19m23.828783613s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025719000 (3h19m23.829078184s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025933739 (3h19m23.829292922s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026092417 (3h19m23.829451591s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: write tcp 159.223.143.39:50068->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026120182 (3h19m23.829479349s) [seg_pos={0 0}, upload=3409]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:35:23.049067071 (3h19m23.852426247s) [upload=3409]: Done uploading segments...
Feb 17 23:35:23.049115322 (3h19m23.852474496s) [upload=3409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1.035536249s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
20 out of 60 is failed (threshold=3) [..........X..XX.XX.X...XX.X..XXXXX.XX.X.X..XX...............] duration:
942.534208ms trace:
https://jaeger.dev.storj.io/trace/2be28be9b880734a
FAIL (2026-02-17 23:46:33 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Feb 17 23:34:23.059166009 (3h18m23.862525184s) [upload=3409]: Starting upload
Feb 17 23:34:23.059410193 (3h18m23.862769371s) [upload=3409]: Done starting upload
Feb 17 23:34:23.059535700 (3h18m23.86289487s) [upload=3409]: Uploading segments...
Feb 17 23:34:23.060262272 (3h18m23.863621456s) [upload=3409]: Got next segment. Inline: false
Feb 17 23:34:23.060415851 (3h18m23.86377504s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:34:23.060424507 (3h18m23.863783691s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:34:23.091662596 (3h18m23.89502178s) [seg_pos={0 0}, upload=3409]: Begin upload segment...
Feb 17 23:34:23.091736818 (3h18m23.895096053s) [seg_pos={0 0}, upload=3409]: Done begin upload segment.
Feb 17 23:34:23.091776467 (3h18m23.895135652s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.091923016 (3h18m23.8952822s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.092846932 (3h18m23.896206109s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093631811 (3h18m23.896990994s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093929468 (3h18m23.897288656s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.094936563 (3h18m23.898295747s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.095529634 (3h18m23.898888818s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096283238 (3h18m23.899642414s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096313780 (3h18m23.899672964s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096989058 (3h18m23.900348238s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097111179 (3h18m23.900470354s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097545617 (3h18m23.90090479s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097826531 (3h18m23.90118571s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098091522 (3h18m23.901450698s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098582559 (3h18m23.901941739s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098625393 (3h18m23.901984578s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099438197 (3h18m23.902797382s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099473251 (3h18m23.902832435s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.168134648 (3h18m23.971493869s) [upload=3409]: Next returned nil segment
Feb 17 23:34:23.168173410 (3h18m23.97153259s) [upload=3409]: Waiting for error group managing segments...
Feb 17 23:34:23.192662134 (3h18m23.996021325s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.199821483 (3h18m24.003180644s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.216892202 (3h18m24.020251384s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.315136137 (3h18m24.118495316s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.369180508 (3h18m24.17253969s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.676435083 (3h18m24.479794257s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.686494858 (3h18m24.48985402s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690268874 (3h18m24.493628038s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690325658 (3h18m24.493684832s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.702724806 (3h18m24.506083983s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.704052843 (3h18m24.507412028s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:24.098092821 (3h18m24.901451996s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:35:23.025077135 (3h19m23.82843643s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: rpc: dial tcp 77.90.31.15:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025124657 (3h19m23.828483841s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025424430 (3h19m23.828783613s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025719000 (3h19m23.829078184s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025933739 (3h19m23.829292922s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026092417 (3h19m23.829451591s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: write tcp 159.223.143.39:50068->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026120182 (3h19m23.829479349s) [seg_pos={0 0}, upload=3409]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:35:23.049067071 (3h19m23.852426247s) [upload=3409]: Done uploading segments...
Feb 17 23:35:23.049115322 (3h19m23.852474496s) [upload=3409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 942.577616ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
655.322651ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:46:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 655.427274ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [..........X..X.XX.XXXXX.X...XXXXXXXXX..XX.XX................] duration:
1.238259704s trace:
https://jaeger.dev.storj.io/trace/155330f7fc0e4dbc
FAIL (2026-02-17 23:46:13 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7 23:33:35.296364403 (3h17m36.099723572s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:35.296369828 (3h17m36.099729008s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:35.328531581 (3h17m36.131890757s) [seg_pos={0 0}, upload=3400]: Begin upload segment...
Feb 17 23:33:35.328594851 (3h17m36.131954034s) [seg_pos={0 0}, upload=3400]: Done begin upload segment.
Feb 17 23:33:35.328629116 (3h17m36.131988303s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.329707651 (3h17m36.133066835s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330219639 (3h17m36.133578819s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330709168 (3h17m36.134068348s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331183887 (3h17m36.134543064s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331616342 (3h17m36.134975519s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332168121 (3h17m36.135527298s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332636066 (3h17m36.135995242s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333221709 (3h17m36.136580889s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333738059 (3h17m36.137097242s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334246293 (3h17m36.13760547s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334948323 (3h17m36.138307498s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.335613706 (3h17m36.138972889s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337192803 (3h17m36.140551986s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337747107 (3h17m36.141106281s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338369048 (3h17m36.141728224s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338990672 (3h17m36.142349846s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.339653294 (3h17m36.143012469s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.408953805 (3h17m36.212312973s) [upload=3400]: Next returned nil segment
Feb 17 23:33:35.408965890 (3h17m36.212325065s) [upload=3400]: Waiting for error group managing segments...
Feb 17 23:33:35.418290309 (3h17m36.221649484s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.425756540 (3h17m36.229115704s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.427489628 (3h17m36.230848804s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.461340071 (3h17m36.264699245s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.462428268 (3h17m36.265787445s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.488391421 (3h17m36.291750584s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.923338747 (3h17m36.726697923s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.925771195 (3h17m36.72913037s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.926498548 (3h17m36.729857723s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.966840449 (3h17m36.770199625s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.027376222 (3h17m36.830735406s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.338385603 (3h17m37.14174478s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:34:35.263290924 (3h18m36.066650107s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263414921 (3h18m36.066774097s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12iLnZ6bvgFWKtP79m7pa6rmStrpMGJQeUuDV3kTHQdFZQAWS4, address:77.90.31.16:4752): protocol: write tcp 159.223.143.39:56974->77.90.31.16:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263594757 (3h18m36.066953935s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:41266->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263729993 (3h18m36.067089176s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263772396 (3h18m36.067131576s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: write tcp 159.223.143.39:45336->77.90.31.11:4562: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264125641 (3h18m36.067484825s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: rpc: dial tcp 77.90.31.15:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264162026 (3h18m36.0675212s) [seg_pos={0 0}, upload=3400]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:35.290242339 (3h18m36.09360153s) [upload=3400]: Done uploading segments...
Feb 17 23:34:35.290317435 (3h18m36.093676616s) [upload=3400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1.238293162s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......XX...................................................] duration:
432.865431ms trace:
https://jaeger.dev.storj.io/trace/732e062283542b6f
FAIL (2026-02-17 23:45:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 432.895992ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
657.070677ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:45:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 657.215778ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
20 out of 60 is failed (threshold=3) [...........X..XX.XX.X...XX.X..XXXXX.XX.X.X..XX..............] duration:
473.583738ms trace:
https://jaeger.dev.storj.io/trace/2be28be9b880734a
FAIL (2026-02-17 23:45:26 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Feb 17 23:34:23.059166009 (3h18m23.862525184s) [upload=3409]: Starting upload
Feb 17 23:34:23.059410193 (3h18m23.862769371s) [upload=3409]: Done starting upload
Feb 17 23:34:23.059535700 (3h18m23.86289487s) [upload=3409]: Uploading segments...
Feb 17 23:34:23.060262272 (3h18m23.863621456s) [upload=3409]: Got next segment. Inline: false
Feb 17 23:34:23.060415851 (3h18m23.86377504s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:34:23.060424507 (3h18m23.863783691s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:34:23.091662596 (3h18m23.89502178s) [seg_pos={0 0}, upload=3409]: Begin upload segment...
Feb 17 23:34:23.091736818 (3h18m23.895096053s) [seg_pos={0 0}, upload=3409]: Done begin upload segment.
Feb 17 23:34:23.091776467 (3h18m23.895135652s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.091923016 (3h18m23.8952822s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.092846932 (3h18m23.896206109s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093631811 (3h18m23.896990994s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093929468 (3h18m23.897288656s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.094936563 (3h18m23.898295747s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.095529634 (3h18m23.898888818s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096283238 (3h18m23.899642414s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096313780 (3h18m23.899672964s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096989058 (3h18m23.900348238s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097111179 (3h18m23.900470354s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097545617 (3h18m23.90090479s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097826531 (3h18m23.90118571s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098091522 (3h18m23.901450698s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098582559 (3h18m23.901941739s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098625393 (3h18m23.901984578s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099438197 (3h18m23.902797382s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099473251 (3h18m23.902832435s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.168134648 (3h18m23.971493869s) [upload=3409]: Next returned nil segment
Feb 17 23:34:23.168173410 (3h18m23.97153259s) [upload=3409]: Waiting for error group managing segments...
Feb 17 23:34:23.192662134 (3h18m23.996021325s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.199821483 (3h18m24.003180644s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.216892202 (3h18m24.020251384s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.315136137 (3h18m24.118495316s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.369180508 (3h18m24.17253969s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.676435083 (3h18m24.479794257s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.686494858 (3h18m24.48985402s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690268874 (3h18m24.493628038s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690325658 (3h18m24.493684832s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.702724806 (3h18m24.506083983s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.704052843 (3h18m24.507412028s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:24.098092821 (3h18m24.901451996s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:35:23.025077135 (3h19m23.82843643s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: rpc: dial tcp 77.90.31.15:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025124657 (3h19m23.828483841s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025424430 (3h19m23.828783613s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025719000 (3h19m23.829078184s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025933739 (3h19m23.829292922s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026092417 (3h19m23.829451591s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: write tcp 159.223.143.39:50068->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026120182 (3h19m23.829479349s) [seg_pos={0 0}, upload=3409]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:35:23.049067071 (3h19m23.852426247s) [upload=3409]: Done uploading segments...
Feb 17 23:35:23.049115322 (3h19m23.852474496s) [upload=3409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 473.665417ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [X.X.........................................................] duration:
590.505422ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:45:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 590.565668ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........XX..................................................] duration:
473.169125ms trace:
https://jaeger.dev.storj.io/trace/732e062283542b6f
FAIL (2026-02-17 23:45:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 473.21195ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.X.X........................................................] duration:
740.841489ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:45:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 740.861768ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [...........X..X.XX.XXXXX.X...XXXXXXXXX..XX.XX...............] duration:
1.420170306s trace:
https://jaeger.dev.storj.io/trace/155330f7fc0e4dbc
FAIL (2026-02-17 23:45:02 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7 23:33:35.296364403 (3h17m36.099723572s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:35.296369828 (3h17m36.099729008s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:35.328531581 (3h17m36.131890757s) [seg_pos={0 0}, upload=3400]: Begin upload segment...
Feb 17 23:33:35.328594851 (3h17m36.131954034s) [seg_pos={0 0}, upload=3400]: Done begin upload segment.
Feb 17 23:33:35.328629116 (3h17m36.131988303s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.329707651 (3h17m36.133066835s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330219639 (3h17m36.133578819s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330709168 (3h17m36.134068348s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331183887 (3h17m36.134543064s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331616342 (3h17m36.134975519s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332168121 (3h17m36.135527298s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332636066 (3h17m36.135995242s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333221709 (3h17m36.136580889s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333738059 (3h17m36.137097242s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334246293 (3h17m36.13760547s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334948323 (3h17m36.138307498s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.335613706 (3h17m36.138972889s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337192803 (3h17m36.140551986s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337747107 (3h17m36.141106281s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338369048 (3h17m36.141728224s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338990672 (3h17m36.142349846s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.339653294 (3h17m36.143012469s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.408953805 (3h17m36.212312973s) [upload=3400]: Next returned nil segment
Feb 17 23:33:35.408965890 (3h17m36.212325065s) [upload=3400]: Waiting for error group managing segments...
Feb 17 23:33:35.418290309 (3h17m36.221649484s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.425756540 (3h17m36.229115704s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.427489628 (3h17m36.230848804s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.461340071 (3h17m36.264699245s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.462428268 (3h17m36.265787445s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.488391421 (3h17m36.291750584s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.923338747 (3h17m36.726697923s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.925771195 (3h17m36.72913037s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.926498548 (3h17m36.729857723s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.966840449 (3h17m36.770199625s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.027376222 (3h17m36.830735406s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.338385603 (3h17m37.14174478s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:34:35.263290924 (3h18m36.066650107s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263414921 (3h18m36.066774097s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12iLnZ6bvgFWKtP79m7pa6rmStrpMGJQeUuDV3kTHQdFZQAWS4, address:77.90.31.16:4752): protocol: write tcp 159.223.143.39:56974->77.90.31.16:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263594757 (3h18m36.066953935s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:41266->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263729993 (3h18m36.067089176s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263772396 (3h18m36.067131576s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: write tcp 159.223.143.39:45336->77.90.31.11:4562: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264125641 (3h18m36.067484825s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: rpc: dial tcp 77.90.31.15:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264162026 (3h18m36.0675212s) [seg_pos={0 0}, upload=3400]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:35.290242339 (3h18m36.09360153s) [upload=3400]: Done uploading segments...
Feb 17 23:34:35.290317435 (3h18m36.093676616s) [upload=3400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1.420206236s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [............X..X.XX.XXXXX.X...XXXXXXXXX..XX.XX..............] duration:
968.087524ms trace:
https://jaeger.dev.storj.io/trace/155330f7fc0e4dbc
FAIL (2026-02-17 23:44:44 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7 23:33:35.296364403 (3h17m36.099723572s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:35.296369828 (3h17m36.099729008s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:35.328531581 (3h17m36.131890757s) [seg_pos={0 0}, upload=3400]: Begin upload segment...
Feb 17 23:33:35.328594851 (3h17m36.131954034s) [seg_pos={0 0}, upload=3400]: Done begin upload segment.
Feb 17 23:33:35.328629116 (3h17m36.131988303s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.329707651 (3h17m36.133066835s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330219639 (3h17m36.133578819s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330709168 (3h17m36.134068348s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331183887 (3h17m36.134543064s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331616342 (3h17m36.134975519s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332168121 (3h17m36.135527298s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332636066 (3h17m36.135995242s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333221709 (3h17m36.136580889s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333738059 (3h17m36.137097242s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334246293 (3h17m36.13760547s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334948323 (3h17m36.138307498s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.335613706 (3h17m36.138972889s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337192803 (3h17m36.140551986s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337747107 (3h17m36.141106281s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338369048 (3h17m36.141728224s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338990672 (3h17m36.142349846s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.339653294 (3h17m36.143012469s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.408953805 (3h17m36.212312973s) [upload=3400]: Next returned nil segment
Feb 17 23:33:35.408965890 (3h17m36.212325065s) [upload=3400]: Waiting for error group managing segments...
Feb 17 23:33:35.418290309 (3h17m36.221649484s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.425756540 (3h17m36.229115704s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.427489628 (3h17m36.230848804s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.461340071 (3h17m36.264699245s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.462428268 (3h17m36.265787445s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.488391421 (3h17m36.291750584s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.923338747 (3h17m36.726697923s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.925771195 (3h17m36.72913037s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.926498548 (3h17m36.729857723s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.966840449 (3h17m36.770199625s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.027376222 (3h17m36.830735406s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.338385603 (3h17m37.14174478s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:34:35.263290924 (3h18m36.066650107s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263414921 (3h18m36.066774097s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12iLnZ6bvgFWKtP79m7pa6rmStrpMGJQeUuDV3kTHQdFZQAWS4, address:77.90.31.16:4752): protocol: write tcp 159.223.143.39:56974->77.90.31.16:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263594757 (3h18m36.066953935s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:41266->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263729993 (3h18m36.067089176s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263772396 (3h18m36.067131576s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: write tcp 159.223.143.39:45336->77.90.31.11:4562: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264125641 (3h18m36.067484825s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: rpc: dial tcp 77.90.31.15:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264162026 (3h18m36.0675212s) [seg_pos={0 0}, upload=3400]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:35.290242339 (3h18m36.09360153s) [upload=3400]: Done uploading segments...
Feb 17 23:34:35.290317435 (3h18m36.093676616s) [upload=3400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 968.122123ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
20 out of 60 is failed (threshold=3) [............X..XX.XX.X...XX.X..XXXXX.XX.X.X..XX.............] duration:
443.816306ms trace:
https://jaeger.dev.storj.io/trace/2be28be9b880734a
FAIL (2026-02-17 23:44:43 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Feb 17 23:34:23.059166009 (3h18m23.862525184s) [upload=3409]: Starting upload
Feb 17 23:34:23.059410193 (3h18m23.862769371s) [upload=3409]: Done starting upload
Feb 17 23:34:23.059535700 (3h18m23.86289487s) [upload=3409]: Uploading segments...
Feb 17 23:34:23.060262272 (3h18m23.863621456s) [upload=3409]: Got next segment. Inline: false
Feb 17 23:34:23.060415851 (3h18m23.86377504s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:34:23.060424507 (3h18m23.863783691s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:34:23.091662596 (3h18m23.89502178s) [seg_pos={0 0}, upload=3409]: Begin upload segment...
Feb 17 23:34:23.091736818 (3h18m23.895096053s) [seg_pos={0 0}, upload=3409]: Done begin upload segment.
Feb 17 23:34:23.091776467 (3h18m23.895135652s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.091923016 (3h18m23.8952822s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.092846932 (3h18m23.896206109s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093631811 (3h18m23.896990994s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093929468 (3h18m23.897288656s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.094936563 (3h18m23.898295747s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.095529634 (3h18m23.898888818s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096283238 (3h18m23.899642414s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096313780 (3h18m23.899672964s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096989058 (3h18m23.900348238s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097111179 (3h18m23.900470354s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097545617 (3h18m23.90090479s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097826531 (3h18m23.90118571s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098091522 (3h18m23.901450698s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098582559 (3h18m23.901941739s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098625393 (3h18m23.901984578s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099438197 (3h18m23.902797382s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099473251 (3h18m23.902832435s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.168134648 (3h18m23.971493869s) [upload=3409]: Next returned nil segment
Feb 17 23:34:23.168173410 (3h18m23.97153259s) [upload=3409]: Waiting for error group managing segments...
Feb 17 23:34:23.192662134 (3h18m23.996021325s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.199821483 (3h18m24.003180644s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.216892202 (3h18m24.020251384s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.315136137 (3h18m24.118495316s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.369180508 (3h18m24.17253969s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.676435083 (3h18m24.479794257s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.686494858 (3h18m24.48985402s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690268874 (3h18m24.493628038s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690325658 (3h18m24.493684832s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.702724806 (3h18m24.506083983s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.704052843 (3h18m24.507412028s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:24.098092821 (3h18m24.901451996s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:35:23.025077135 (3h19m23.82843643s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: rpc: dial tcp 77.90.31.15:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025124657 (3h19m23.828483841s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025424430 (3h19m23.828783613s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025719000 (3h19m23.829078184s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025933739 (3h19m23.829292922s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026092417 (3h19m23.829451591s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: write tcp 159.223.143.39:50068->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026120182 (3h19m23.829479349s) [seg_pos={0 0}, upload=3409]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:35:23.049067071 (3h19m23.852426247s) [upload=3409]: Done uploading segments...
Feb 17 23:35:23.049115322 (3h19m23.852474496s) [upload=3409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 443.839121ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..X.X.......................................................] duration:
578.353498ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:44:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 578.380273ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...X.X......................................................] duration:
592.284431ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:44:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 592.308062ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........XX.................................................] duration:
454.467992ms trace:
https://jaeger.dev.storj.io/trace/732e062283542b6f
FAIL (2026-02-17 23:44:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 454.517208ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [.............X..X.XX.XXXXX.X...XXXXXXXXX..XX.XX.............] duration:
918.141736ms trace:
https://jaeger.dev.storj.io/trace/155330f7fc0e4dbc
FAIL (2026-02-17 23:43:59 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7 23:33:35.296364403 (3h17m36.099723572s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:35.296369828 (3h17m36.099729008s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:35.328531581 (3h17m36.131890757s) [seg_pos={0 0}, upload=3400]: Begin upload segment...
Feb 17 23:33:35.328594851 (3h17m36.131954034s) [seg_pos={0 0}, upload=3400]: Done begin upload segment.
Feb 17 23:33:35.328629116 (3h17m36.131988303s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.329707651 (3h17m36.133066835s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330219639 (3h17m36.133578819s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330709168 (3h17m36.134068348s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331183887 (3h17m36.134543064s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331616342 (3h17m36.134975519s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332168121 (3h17m36.135527298s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332636066 (3h17m36.135995242s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333221709 (3h17m36.136580889s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333738059 (3h17m36.137097242s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334246293 (3h17m36.13760547s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334948323 (3h17m36.138307498s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.335613706 (3h17m36.138972889s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337192803 (3h17m36.140551986s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337747107 (3h17m36.141106281s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338369048 (3h17m36.141728224s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338990672 (3h17m36.142349846s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.339653294 (3h17m36.143012469s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.408953805 (3h17m36.212312973s) [upload=3400]: Next returned nil segment
Feb 17 23:33:35.408965890 (3h17m36.212325065s) [upload=3400]: Waiting for error group managing segments...
Feb 17 23:33:35.418290309 (3h17m36.221649484s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.425756540 (3h17m36.229115704s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.427489628 (3h17m36.230848804s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.461340071 (3h17m36.264699245s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.462428268 (3h17m36.265787445s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.488391421 (3h17m36.291750584s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.923338747 (3h17m36.726697923s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.925771195 (3h17m36.72913037s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.926498548 (3h17m36.729857723s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.966840449 (3h17m36.770199625s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.027376222 (3h17m36.830735406s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.338385603 (3h17m37.14174478s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:34:35.263290924 (3h18m36.066650107s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263414921 (3h18m36.066774097s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12iLnZ6bvgFWKtP79m7pa6rmStrpMGJQeUuDV3kTHQdFZQAWS4, address:77.90.31.16:4752): protocol: write tcp 159.223.143.39:56974->77.90.31.16:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263594757 (3h18m36.066953935s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:41266->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263729993 (3h18m36.067089176s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263772396 (3h18m36.067131576s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: write tcp 159.223.143.39:45336->77.90.31.11:4562: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264125641 (3h18m36.067484825s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: rpc: dial tcp 77.90.31.15:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264162026 (3h18m36.0675212s) [seg_pos={0 0}, upload=3400]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:35.290242339 (3h18m36.09360153s) [upload=3400]: Done uploading segments...
Feb 17 23:34:35.290317435 (3h18m36.093676616s) [upload=3400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 918.166388ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [..............X..X.XX.XXXXX.X...XXXXXXXXX..XX.XX............] duration:
886.837158ms trace:
https://jaeger.dev.storj.io/trace/155330f7fc0e4dbc
FAIL (2026-02-17 23:43:38 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7 23:33:35.296364403 (3h17m36.099723572s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:35.296369828 (3h17m36.099729008s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:35.328531581 (3h17m36.131890757s) [seg_pos={0 0}, upload=3400]: Begin upload segment...
Feb 17 23:33:35.328594851 (3h17m36.131954034s) [seg_pos={0 0}, upload=3400]: Done begin upload segment.
Feb 17 23:33:35.328629116 (3h17m36.131988303s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.329707651 (3h17m36.133066835s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330219639 (3h17m36.133578819s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330709168 (3h17m36.134068348s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331183887 (3h17m36.134543064s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331616342 (3h17m36.134975519s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332168121 (3h17m36.135527298s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332636066 (3h17m36.135995242s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333221709 (3h17m36.136580889s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333738059 (3h17m36.137097242s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334246293 (3h17m36.13760547s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334948323 (3h17m36.138307498s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.335613706 (3h17m36.138972889s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337192803 (3h17m36.140551986s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337747107 (3h17m36.141106281s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338369048 (3h17m36.141728224s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338990672 (3h17m36.142349846s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.339653294 (3h17m36.143012469s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.408953805 (3h17m36.212312973s) [upload=3400]: Next returned nil segment
Feb 17 23:33:35.408965890 (3h17m36.212325065s) [upload=3400]: Waiting for error group managing segments...
Feb 17 23:33:35.418290309 (3h17m36.221649484s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.425756540 (3h17m36.229115704s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.427489628 (3h17m36.230848804s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.461340071 (3h17m36.264699245s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.462428268 (3h17m36.265787445s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.488391421 (3h17m36.291750584s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.923338747 (3h17m36.726697923s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.925771195 (3h17m36.72913037s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.926498548 (3h17m36.729857723s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.966840449 (3h17m36.770199625s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.027376222 (3h17m36.830735406s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.338385603 (3h17m37.14174478s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:34:35.263290924 (3h18m36.066650107s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263414921 (3h18m36.066774097s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12iLnZ6bvgFWKtP79m7pa6rmStrpMGJQeUuDV3kTHQdFZQAWS4, address:77.90.31.16:4752): protocol: write tcp 159.223.143.39:56974->77.90.31.16:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263594757 (3h18m36.066953935s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:41266->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263729993 (3h18m36.067089176s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263772396 (3h18m36.067131576s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: write tcp 159.223.143.39:45336->77.90.31.11:4562: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264125641 (3h18m36.067484825s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: rpc: dial tcp 77.90.31.15:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264162026 (3h18m36.0675212s) [seg_pos={0 0}, upload=3400]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:35.290242339 (3h18m36.09360153s) [upload=3400]: Done uploading segments...
Feb 17 23:34:35.290317435 (3h18m36.093676616s) [upload=3400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 886.861267ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
20 out of 60 is failed (threshold=3) [.............X..XX.XX.X...XX.X..XXXXX.XX.X.X..XX............] duration:
1.193608044s trace:
https://jaeger.dev.storj.io/trace/2be28be9b880734a
FAIL (2026-02-17 23:43:32 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Feb 17 23:34:23.059166009 (3h18m23.862525184s) [upload=3409]: Starting upload
Feb 17 23:34:23.059410193 (3h18m23.862769371s) [upload=3409]: Done starting upload
Feb 17 23:34:23.059535700 (3h18m23.86289487s) [upload=3409]: Uploading segments...
Feb 17 23:34:23.060262272 (3h18m23.863621456s) [upload=3409]: Got next segment. Inline: false
Feb 17 23:34:23.060415851 (3h18m23.86377504s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:34:23.060424507 (3h18m23.863783691s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:34:23.091662596 (3h18m23.89502178s) [seg_pos={0 0}, upload=3409]: Begin upload segment...
Feb 17 23:34:23.091736818 (3h18m23.895096053s) [seg_pos={0 0}, upload=3409]: Done begin upload segment.
Feb 17 23:34:23.091776467 (3h18m23.895135652s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.091923016 (3h18m23.8952822s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.092846932 (3h18m23.896206109s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093631811 (3h18m23.896990994s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093929468 (3h18m23.897288656s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.094936563 (3h18m23.898295747s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.095529634 (3h18m23.898888818s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096283238 (3h18m23.899642414s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096313780 (3h18m23.899672964s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096989058 (3h18m23.900348238s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097111179 (3h18m23.900470354s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097545617 (3h18m23.90090479s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097826531 (3h18m23.90118571s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098091522 (3h18m23.901450698s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098582559 (3h18m23.901941739s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098625393 (3h18m23.901984578s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099438197 (3h18m23.902797382s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099473251 (3h18m23.902832435s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.168134648 (3h18m23.971493869s) [upload=3409]: Next returned nil segment
Feb 17 23:34:23.168173410 (3h18m23.97153259s) [upload=3409]: Waiting for error group managing segments...
Feb 17 23:34:23.192662134 (3h18m23.996021325s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.199821483 (3h18m24.003180644s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.216892202 (3h18m24.020251384s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.315136137 (3h18m24.118495316s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.369180508 (3h18m24.17253969s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.676435083 (3h18m24.479794257s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.686494858 (3h18m24.48985402s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690268874 (3h18m24.493628038s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690325658 (3h18m24.493684832s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.702724806 (3h18m24.506083983s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.704052843 (3h18m24.507412028s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:24.098092821 (3h18m24.901451996s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:35:23.025077135 (3h19m23.82843643s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: rpc: dial tcp 77.90.31.15:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025124657 (3h19m23.828483841s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025424430 (3h19m23.828783613s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025719000 (3h19m23.829078184s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025933739 (3h19m23.829292922s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026092417 (3h19m23.829451591s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: write tcp 159.223.143.39:50068->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026120182 (3h19m23.829479349s) [seg_pos={0 0}, upload=3409]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:35:23.049067071 (3h19m23.852426247s) [upload=3409]: Done uploading segments...
Feb 17 23:35:23.049115322 (3h19m23.852474496s) [upload=3409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1.193646921s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........XX................................................] duration:
522.951533ms trace:
https://jaeger.dev.storj.io/trace/732e062283542b6f
FAIL (2026-02-17 23:43:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 522.989547ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....X.X.....................................................] duration:
908.986273ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:43:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 909.056987ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [...............X..X.XX.XXXXX.X...XXXXXXXXX..XX.XX...........] duration:
930.891551ms trace:
https://jaeger.dev.storj.io/trace/155330f7fc0e4dbc
FAIL (2026-02-17 23:42:44 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7 23:33:35.296364403 (3h17m36.099723572s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:35.296369828 (3h17m36.099729008s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:35.328531581 (3h17m36.131890757s) [seg_pos={0 0}, upload=3400]: Begin upload segment...
Feb 17 23:33:35.328594851 (3h17m36.131954034s) [seg_pos={0 0}, upload=3400]: Done begin upload segment.
Feb 17 23:33:35.328629116 (3h17m36.131988303s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.329707651 (3h17m36.133066835s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330219639 (3h17m36.133578819s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330709168 (3h17m36.134068348s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331183887 (3h17m36.134543064s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331616342 (3h17m36.134975519s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332168121 (3h17m36.135527298s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332636066 (3h17m36.135995242s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333221709 (3h17m36.136580889s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333738059 (3h17m36.137097242s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334246293 (3h17m36.13760547s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334948323 (3h17m36.138307498s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.335613706 (3h17m36.138972889s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337192803 (3h17m36.140551986s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337747107 (3h17m36.141106281s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338369048 (3h17m36.141728224s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338990672 (3h17m36.142349846s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.339653294 (3h17m36.143012469s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.408953805 (3h17m36.212312973s) [upload=3400]: Next returned nil segment
Feb 17 23:33:35.408965890 (3h17m36.212325065s) [upload=3400]: Waiting for error group managing segments...
Feb 17 23:33:35.418290309 (3h17m36.221649484s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.425756540 (3h17m36.229115704s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.427489628 (3h17m36.230848804s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.461340071 (3h17m36.264699245s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.462428268 (3h17m36.265787445s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.488391421 (3h17m36.291750584s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.923338747 (3h17m36.726697923s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.925771195 (3h17m36.72913037s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.926498548 (3h17m36.729857723s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.966840449 (3h17m36.770199625s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.027376222 (3h17m36.830735406s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.338385603 (3h17m37.14174478s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:34:35.263290924 (3h18m36.066650107s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263414921 (3h18m36.066774097s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12iLnZ6bvgFWKtP79m7pa6rmStrpMGJQeUuDV3kTHQdFZQAWS4, address:77.90.31.16:4752): protocol: write tcp 159.223.143.39:56974->77.90.31.16:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263594757 (3h18m36.066953935s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:41266->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263729993 (3h18m36.067089176s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263772396 (3h18m36.067131576s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: write tcp 159.223.143.39:45336->77.90.31.11:4562: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264125641 (3h18m36.067484825s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: rpc: dial tcp 77.90.31.15:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264162026 (3h18m36.0675212s) [seg_pos={0 0}, upload=3400]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:35.290242339 (3h18m36.09360153s) [upload=3400]: Done uploading segments...
Feb 17 23:34:35.290317435 (3h18m36.093676616s) [upload=3400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 930.919052ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
20 out of 60 is failed (threshold=3) [..............X..XX.XX.X...XX.X..XXXXX.XX.X.X..XX...........] duration:
861.223239ms trace:
https://jaeger.dev.storj.io/trace/2be28be9b880734a
FAIL (2026-02-17 23:42:34 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Feb 17 23:34:23.059166009 (3h18m23.862525184s) [upload=3409]: Starting upload
Feb 17 23:34:23.059410193 (3h18m23.862769371s) [upload=3409]: Done starting upload
Feb 17 23:34:23.059535700 (3h18m23.86289487s) [upload=3409]: Uploading segments...
Feb 17 23:34:23.060262272 (3h18m23.863621456s) [upload=3409]: Got next segment. Inline: false
Feb 17 23:34:23.060415851 (3h18m23.86377504s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:34:23.060424507 (3h18m23.863783691s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:34:23.091662596 (3h18m23.89502178s) [seg_pos={0 0}, upload=3409]: Begin upload segment...
Feb 17 23:34:23.091736818 (3h18m23.895096053s) [seg_pos={0 0}, upload=3409]: Done begin upload segment.
Feb 17 23:34:23.091776467 (3h18m23.895135652s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.091923016 (3h18m23.8952822s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.092846932 (3h18m23.896206109s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093631811 (3h18m23.896990994s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093929468 (3h18m23.897288656s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.094936563 (3h18m23.898295747s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.095529634 (3h18m23.898888818s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096283238 (3h18m23.899642414s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096313780 (3h18m23.899672964s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096989058 (3h18m23.900348238s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097111179 (3h18m23.900470354s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097545617 (3h18m23.90090479s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097826531 (3h18m23.90118571s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098091522 (3h18m23.901450698s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098582559 (3h18m23.901941739s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098625393 (3h18m23.901984578s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099438197 (3h18m23.902797382s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099473251 (3h18m23.902832435s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.168134648 (3h18m23.971493869s) [upload=3409]: Next returned nil segment
Feb 17 23:34:23.168173410 (3h18m23.97153259s) [upload=3409]: Waiting for error group managing segments...
Feb 17 23:34:23.192662134 (3h18m23.996021325s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.199821483 (3h18m24.003180644s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.216892202 (3h18m24.020251384s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.315136137 (3h18m24.118495316s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.369180508 (3h18m24.17253969s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.676435083 (3h18m24.479794257s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.686494858 (3h18m24.48985402s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690268874 (3h18m24.493628038s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690325658 (3h18m24.493684832s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.702724806 (3h18m24.506083983s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.704052843 (3h18m24.507412028s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:24.098092821 (3h18m24.901451996s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:35:23.025077135 (3h19m23.82843643s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: rpc: dial tcp 77.90.31.15:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025124657 (3h19m23.828483841s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025424430 (3h19m23.828783613s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025719000 (3h19m23.829078184s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025933739 (3h19m23.829292922s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026092417 (3h19m23.829451591s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: write tcp 159.223.143.39:50068->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026120182 (3h19m23.829479349s) [seg_pos={0 0}, upload=3409]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:35:23.049067071 (3h19m23.852426247s) [upload=3409]: Done uploading segments...
Feb 17 23:35:23.049115322 (3h19m23.852474496s) [upload=3409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 861.251713ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....X.X....................................................] duration:
568.969279ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:42:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 569.000099ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [................X..X.XX.XXXXX.X...XXXXXXXXX..XX.XX..........] duration:
997.456135ms trace:
https://jaeger.dev.storj.io/trace/155330f7fc0e4dbc
FAIL (2026-02-17 23:42:20 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7 23:33:35.296364403 (3h17m36.099723572s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:35.296369828 (3h17m36.099729008s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:35.328531581 (3h17m36.131890757s) [seg_pos={0 0}, upload=3400]: Begin upload segment...
Feb 17 23:33:35.328594851 (3h17m36.131954034s) [seg_pos={0 0}, upload=3400]: Done begin upload segment.
Feb 17 23:33:35.328629116 (3h17m36.131988303s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.329707651 (3h17m36.133066835s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330219639 (3h17m36.133578819s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330709168 (3h17m36.134068348s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331183887 (3h17m36.134543064s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331616342 (3h17m36.134975519s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332168121 (3h17m36.135527298s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332636066 (3h17m36.135995242s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333221709 (3h17m36.136580889s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333738059 (3h17m36.137097242s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334246293 (3h17m36.13760547s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334948323 (3h17m36.138307498s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.335613706 (3h17m36.138972889s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337192803 (3h17m36.140551986s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337747107 (3h17m36.141106281s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338369048 (3h17m36.141728224s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338990672 (3h17m36.142349846s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.339653294 (3h17m36.143012469s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.408953805 (3h17m36.212312973s) [upload=3400]: Next returned nil segment
Feb 17 23:33:35.408965890 (3h17m36.212325065s) [upload=3400]: Waiting for error group managing segments...
Feb 17 23:33:35.418290309 (3h17m36.221649484s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.425756540 (3h17m36.229115704s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.427489628 (3h17m36.230848804s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.461340071 (3h17m36.264699245s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.462428268 (3h17m36.265787445s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.488391421 (3h17m36.291750584s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.923338747 (3h17m36.726697923s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.925771195 (3h17m36.72913037s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.926498548 (3h17m36.729857723s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.966840449 (3h17m36.770199625s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.027376222 (3h17m36.830735406s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.338385603 (3h17m37.14174478s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:34:35.263290924 (3h18m36.066650107s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263414921 (3h18m36.066774097s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12iLnZ6bvgFWKtP79m7pa6rmStrpMGJQeUuDV3kTHQdFZQAWS4, address:77.90.31.16:4752): protocol: write tcp 159.223.143.39:56974->77.90.31.16:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263594757 (3h18m36.066953935s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:41266->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263729993 (3h18m36.067089176s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263772396 (3h18m36.067131576s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: write tcp 159.223.143.39:45336->77.90.31.11:4562: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264125641 (3h18m36.067484825s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: rpc: dial tcp 77.90.31.15:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264162026 (3h18m36.0675212s) [seg_pos={0 0}, upload=3400]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:35.290242339 (3h18m36.09360153s) [upload=3400]: Done uploading segments...
Feb 17 23:34:35.290317435 (3h18m36.093676616s) [upload=3400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 997.499853ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...........XX...............................................] duration:
465.384833ms trace:
https://jaeger.dev.storj.io/trace/732e062283542b6f
FAIL (2026-02-17 23:42:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 465.411214ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......X.X...................................................] duration:
577.429635ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:42:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 577.467252ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [............XX..............................................] duration:
445.487082ms trace:
https://jaeger.dev.storj.io/trace/732e062283542b6f
FAIL (2026-02-17 23:41:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 445.529409ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
20 out of 60 is failed (threshold=3) [...............X..XX.XX.X...XX.X..XXXXX.XX.X.X..XX..........] duration:
862.823405ms trace:
https://jaeger.dev.storj.io/trace/2be28be9b880734a
FAIL (2026-02-17 23:41:27 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Feb 17 23:34:23.059166009 (3h18m23.862525184s) [upload=3409]: Starting upload
Feb 17 23:34:23.059410193 (3h18m23.862769371s) [upload=3409]: Done starting upload
Feb 17 23:34:23.059535700 (3h18m23.86289487s) [upload=3409]: Uploading segments...
Feb 17 23:34:23.060262272 (3h18m23.863621456s) [upload=3409]: Got next segment. Inline: false
Feb 17 23:34:23.060415851 (3h18m23.86377504s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:34:23.060424507 (3h18m23.863783691s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:34:23.091662596 (3h18m23.89502178s) [seg_pos={0 0}, upload=3409]: Begin upload segment...
Feb 17 23:34:23.091736818 (3h18m23.895096053s) [seg_pos={0 0}, upload=3409]: Done begin upload segment.
Feb 17 23:34:23.091776467 (3h18m23.895135652s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.091923016 (3h18m23.8952822s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.092846932 (3h18m23.896206109s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093631811 (3h18m23.896990994s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093929468 (3h18m23.897288656s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.094936563 (3h18m23.898295747s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.095529634 (3h18m23.898888818s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096283238 (3h18m23.899642414s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096313780 (3h18m23.899672964s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096989058 (3h18m23.900348238s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097111179 (3h18m23.900470354s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097545617 (3h18m23.90090479s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097826531 (3h18m23.90118571s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098091522 (3h18m23.901450698s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098582559 (3h18m23.901941739s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098625393 (3h18m23.901984578s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099438197 (3h18m23.902797382s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099473251 (3h18m23.902832435s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.168134648 (3h18m23.971493869s) [upload=3409]: Next returned nil segment
Feb 17 23:34:23.168173410 (3h18m23.97153259s) [upload=3409]: Waiting for error group managing segments...
Feb 17 23:34:23.192662134 (3h18m23.996021325s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.199821483 (3h18m24.003180644s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.216892202 (3h18m24.020251384s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.315136137 (3h18m24.118495316s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.369180508 (3h18m24.17253969s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.676435083 (3h18m24.479794257s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.686494858 (3h18m24.48985402s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690268874 (3h18m24.493628038s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690325658 (3h18m24.493684832s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.702724806 (3h18m24.506083983s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.704052843 (3h18m24.507412028s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:24.098092821 (3h18m24.901451996s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:35:23.025077135 (3h19m23.82843643s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: rpc: dial tcp 77.90.31.15:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025124657 (3h19m23.828483841s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025424430 (3h19m23.828783613s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025719000 (3h19m23.829078184s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025933739 (3h19m23.829292922s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026092417 (3h19m23.829451591s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: write tcp 159.223.143.39:50068->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026120182 (3h19m23.829479349s) [seg_pos={0 0}, upload=3409]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:35:23.049067071 (3h19m23.852426247s) [upload=3409]: Done uploading segments...
Feb 17 23:35:23.049115322 (3h19m23.852474496s) [upload=3409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 862.850494ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......X.X..................................................] duration:
601.185804ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:41:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 601.216171ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [.................X..X.XX.XXXXX.X...XXXXXXXXX..XX.XX.........] duration:
907.788385ms trace:
https://jaeger.dev.storj.io/trace/155330f7fc0e4dbc
FAIL (2026-02-17 23:41:10 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7 23:33:35.296364403 (3h17m36.099723572s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:35.296369828 (3h17m36.099729008s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:35.328531581 (3h17m36.131890757s) [seg_pos={0 0}, upload=3400]: Begin upload segment...
Feb 17 23:33:35.328594851 (3h17m36.131954034s) [seg_pos={0 0}, upload=3400]: Done begin upload segment.
Feb 17 23:33:35.328629116 (3h17m36.131988303s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.329707651 (3h17m36.133066835s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330219639 (3h17m36.133578819s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330709168 (3h17m36.134068348s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331183887 (3h17m36.134543064s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331616342 (3h17m36.134975519s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332168121 (3h17m36.135527298s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332636066 (3h17m36.135995242s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333221709 (3h17m36.136580889s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333738059 (3h17m36.137097242s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334246293 (3h17m36.13760547s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334948323 (3h17m36.138307498s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.335613706 (3h17m36.138972889s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337192803 (3h17m36.140551986s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337747107 (3h17m36.141106281s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338369048 (3h17m36.141728224s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338990672 (3h17m36.142349846s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.339653294 (3h17m36.143012469s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.408953805 (3h17m36.212312973s) [upload=3400]: Next returned nil segment
Feb 17 23:33:35.408965890 (3h17m36.212325065s) [upload=3400]: Waiting for error group managing segments...
Feb 17 23:33:35.418290309 (3h17m36.221649484s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.425756540 (3h17m36.229115704s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.427489628 (3h17m36.230848804s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.461340071 (3h17m36.264699245s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.462428268 (3h17m36.265787445s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.488391421 (3h17m36.291750584s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.923338747 (3h17m36.726697923s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.925771195 (3h17m36.72913037s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.926498548 (3h17m36.729857723s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.966840449 (3h17m36.770199625s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.027376222 (3h17m36.830735406s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.338385603 (3h17m37.14174478s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:34:35.263290924 (3h18m36.066650107s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263414921 (3h18m36.066774097s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12iLnZ6bvgFWKtP79m7pa6rmStrpMGJQeUuDV3kTHQdFZQAWS4, address:77.90.31.16:4752): protocol: write tcp 159.223.143.39:56974->77.90.31.16:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263594757 (3h18m36.066953935s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:41266->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263729993 (3h18m36.067089176s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263772396 (3h18m36.067131576s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: write tcp 159.223.143.39:45336->77.90.31.11:4562: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264125641 (3h18m36.067484825s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: rpc: dial tcp 77.90.31.15:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264162026 (3h18m36.0675212s) [seg_pos={0 0}, upload=3400]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:35.290242339 (3h18m36.09360153s) [upload=3400]: Done uploading segments...
Feb 17 23:34:35.290317435 (3h18m36.093676616s) [upload=3400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 907.851429ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
20 out of 60 is failed (threshold=3) [................X..XX.XX.X...XX.X..XXXXX.XX.X.X..XX.........] duration:
1.027000702s trace:
https://jaeger.dev.storj.io/trace/2be28be9b880734a
FAIL (2026-02-17 23:41:07 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Feb 17 23:34:23.059166009 (3h18m23.862525184s) [upload=3409]: Starting upload
Feb 17 23:34:23.059410193 (3h18m23.862769371s) [upload=3409]: Done starting upload
Feb 17 23:34:23.059535700 (3h18m23.86289487s) [upload=3409]: Uploading segments...
Feb 17 23:34:23.060262272 (3h18m23.863621456s) [upload=3409]: Got next segment. Inline: false
Feb 17 23:34:23.060415851 (3h18m23.86377504s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:34:23.060424507 (3h18m23.863783691s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:34:23.091662596 (3h18m23.89502178s) [seg_pos={0 0}, upload=3409]: Begin upload segment...
Feb 17 23:34:23.091736818 (3h18m23.895096053s) [seg_pos={0 0}, upload=3409]: Done begin upload segment.
Feb 17 23:34:23.091776467 (3h18m23.895135652s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.091923016 (3h18m23.8952822s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.092846932 (3h18m23.896206109s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093631811 (3h18m23.896990994s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093929468 (3h18m23.897288656s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.094936563 (3h18m23.898295747s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.095529634 (3h18m23.898888818s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096283238 (3h18m23.899642414s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096313780 (3h18m23.899672964s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096989058 (3h18m23.900348238s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097111179 (3h18m23.900470354s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097545617 (3h18m23.90090479s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097826531 (3h18m23.90118571s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098091522 (3h18m23.901450698s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098582559 (3h18m23.901941739s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098625393 (3h18m23.901984578s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099438197 (3h18m23.902797382s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099473251 (3h18m23.902832435s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.168134648 (3h18m23.971493869s) [upload=3409]: Next returned nil segment
Feb 17 23:34:23.168173410 (3h18m23.97153259s) [upload=3409]: Waiting for error group managing segments...
Feb 17 23:34:23.192662134 (3h18m23.996021325s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.199821483 (3h18m24.003180644s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.216892202 (3h18m24.020251384s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.315136137 (3h18m24.118495316s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.369180508 (3h18m24.17253969s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.676435083 (3h18m24.479794257s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.686494858 (3h18m24.48985402s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690268874 (3h18m24.493628038s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690325658 (3h18m24.493684832s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.702724806 (3h18m24.506083983s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.704052843 (3h18m24.507412028s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:24.098092821 (3h18m24.901451996s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:35:23.025077135 (3h19m23.82843643s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: rpc: dial tcp 77.90.31.15:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025124657 (3h19m23.828483841s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025424430 (3h19m23.828783613s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025719000 (3h19m23.829078184s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025933739 (3h19m23.829292922s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026092417 (3h19m23.829451591s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: write tcp 159.223.143.39:50068->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026120182 (3h19m23.829479349s) [seg_pos={0 0}, upload=3409]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:35:23.049067071 (3h19m23.852426247s) [upload=3409]: Done uploading segments...
Feb 17 23:35:23.049115322 (3h19m23.852474496s) [upload=3409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1.027043971s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.............XX.............................................] duration:
541.635738ms trace:
https://jaeger.dev.storj.io/trace/732e062283542b6f
FAIL (2026-02-17 23:41:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 541.702654ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [..................X..X.XX.XXXXX.X...XXXXXXXXX..XX.XX........] duration:
982.366253ms trace:
https://jaeger.dev.storj.io/trace/155330f7fc0e4dbc
FAIL (2026-02-17 23:40:44 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7 23:33:35.296364403 (3h17m36.099723572s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:35.296369828 (3h17m36.099729008s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:35.328531581 (3h17m36.131890757s) [seg_pos={0 0}, upload=3400]: Begin upload segment...
Feb 17 23:33:35.328594851 (3h17m36.131954034s) [seg_pos={0 0}, upload=3400]: Done begin upload segment.
Feb 17 23:33:35.328629116 (3h17m36.131988303s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.329707651 (3h17m36.133066835s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330219639 (3h17m36.133578819s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330709168 (3h17m36.134068348s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331183887 (3h17m36.134543064s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331616342 (3h17m36.134975519s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332168121 (3h17m36.135527298s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332636066 (3h17m36.135995242s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333221709 (3h17m36.136580889s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333738059 (3h17m36.137097242s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334246293 (3h17m36.13760547s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334948323 (3h17m36.138307498s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.335613706 (3h17m36.138972889s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337192803 (3h17m36.140551986s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337747107 (3h17m36.141106281s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338369048 (3h17m36.141728224s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338990672 (3h17m36.142349846s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.339653294 (3h17m36.143012469s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.408953805 (3h17m36.212312973s) [upload=3400]: Next returned nil segment
Feb 17 23:33:35.408965890 (3h17m36.212325065s) [upload=3400]: Waiting for error group managing segments...
Feb 17 23:33:35.418290309 (3h17m36.221649484s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.425756540 (3h17m36.229115704s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.427489628 (3h17m36.230848804s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.461340071 (3h17m36.264699245s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.462428268 (3h17m36.265787445s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.488391421 (3h17m36.291750584s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.923338747 (3h17m36.726697923s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.925771195 (3h17m36.72913037s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.926498548 (3h17m36.729857723s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.966840449 (3h17m36.770199625s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.027376222 (3h17m36.830735406s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.338385603 (3h17m37.14174478s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:34:35.263290924 (3h18m36.066650107s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263414921 (3h18m36.066774097s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12iLnZ6bvgFWKtP79m7pa6rmStrpMGJQeUuDV3kTHQdFZQAWS4, address:77.90.31.16:4752): protocol: write tcp 159.223.143.39:56974->77.90.31.16:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263594757 (3h18m36.066953935s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:41266->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263729993 (3h18m36.067089176s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263772396 (3h18m36.067131576s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: write tcp 159.223.143.39:45336->77.90.31.11:4562: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264125641 (3h18m36.067484825s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: rpc: dial tcp 77.90.31.15:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264162026 (3h18m36.0675212s) [seg_pos={0 0}, upload=3400]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:35.290242339 (3h18m36.09360153s) [upload=3400]: Done uploading segments...
Feb 17 23:34:35.290317435 (3h18m36.093676616s) [upload=3400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 982.396991ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [...................X..X.XX.XXXXX.X...XXXXXXXXX..XX.XX.......] duration:
918.314274ms trace:
https://jaeger.dev.storj.io/trace/155330f7fc0e4dbc
FAIL (2026-02-17 23:40:24 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7 23:33:35.296364403 (3h17m36.099723572s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:35.296369828 (3h17m36.099729008s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:35.328531581 (3h17m36.131890757s) [seg_pos={0 0}, upload=3400]: Begin upload segment...
Feb 17 23:33:35.328594851 (3h17m36.131954034s) [seg_pos={0 0}, upload=3400]: Done begin upload segment.
Feb 17 23:33:35.328629116 (3h17m36.131988303s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.329707651 (3h17m36.133066835s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330219639 (3h17m36.133578819s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330709168 (3h17m36.134068348s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331183887 (3h17m36.134543064s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331616342 (3h17m36.134975519s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332168121 (3h17m36.135527298s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332636066 (3h17m36.135995242s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333221709 (3h17m36.136580889s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333738059 (3h17m36.137097242s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334246293 (3h17m36.13760547s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334948323 (3h17m36.138307498s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.335613706 (3h17m36.138972889s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337192803 (3h17m36.140551986s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337747107 (3h17m36.141106281s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338369048 (3h17m36.141728224s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338990672 (3h17m36.142349846s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.339653294 (3h17m36.143012469s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.408953805 (3h17m36.212312973s) [upload=3400]: Next returned nil segment
Feb 17 23:33:35.408965890 (3h17m36.212325065s) [upload=3400]: Waiting for error group managing segments...
Feb 17 23:33:35.418290309 (3h17m36.221649484s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.425756540 (3h17m36.229115704s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.427489628 (3h17m36.230848804s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.461340071 (3h17m36.264699245s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.462428268 (3h17m36.265787445s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.488391421 (3h17m36.291750584s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.923338747 (3h17m36.726697923s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.925771195 (3h17m36.72913037s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.926498548 (3h17m36.729857723s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.966840449 (3h17m36.770199625s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.027376222 (3h17m36.830735406s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.338385603 (3h17m37.14174478s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:34:35.263290924 (3h18m36.066650107s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263414921 (3h18m36.066774097s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12iLnZ6bvgFWKtP79m7pa6rmStrpMGJQeUuDV3kTHQdFZQAWS4, address:77.90.31.16:4752): protocol: write tcp 159.223.143.39:56974->77.90.31.16:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263594757 (3h18m36.066953935s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:41266->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263729993 (3h18m36.067089176s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263772396 (3h18m36.067131576s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: write tcp 159.223.143.39:45336->77.90.31.11:4562: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264125641 (3h18m36.067484825s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: rpc: dial tcp 77.90.31.15:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264162026 (3h18m36.0675212s) [seg_pos={0 0}, upload=3400]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:35.290242339 (3h18m36.09360153s) [upload=3400]: Done uploading segments...
Feb 17 23:34:35.290317435 (3h18m36.093676616s) [upload=3400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 918.339621ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........X.X.................................................] duration:
585.565517ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:40:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 585.600816ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
20 out of 60 is failed (threshold=3) [.................X..XX.XX.X...XX.X..XXXXX.XX.X.X..XX........] duration:
938.926514ms trace:
https://jaeger.dev.storj.io/trace/2be28be9b880734a
FAIL (2026-02-17 23:40:19 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Feb 17 23:34:23.059166009 (3h18m23.862525184s) [upload=3409]: Starting upload
Feb 17 23:34:23.059410193 (3h18m23.862769371s) [upload=3409]: Done starting upload
Feb 17 23:34:23.059535700 (3h18m23.86289487s) [upload=3409]: Uploading segments...
Feb 17 23:34:23.060262272 (3h18m23.863621456s) [upload=3409]: Got next segment. Inline: false
Feb 17 23:34:23.060415851 (3h18m23.86377504s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:34:23.060424507 (3h18m23.863783691s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:34:23.091662596 (3h18m23.89502178s) [seg_pos={0 0}, upload=3409]: Begin upload segment...
Feb 17 23:34:23.091736818 (3h18m23.895096053s) [seg_pos={0 0}, upload=3409]: Done begin upload segment.
Feb 17 23:34:23.091776467 (3h18m23.895135652s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.091923016 (3h18m23.8952822s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.092846932 (3h18m23.896206109s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093631811 (3h18m23.896990994s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093929468 (3h18m23.897288656s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.094936563 (3h18m23.898295747s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.095529634 (3h18m23.898888818s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096283238 (3h18m23.899642414s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096313780 (3h18m23.899672964s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096989058 (3h18m23.900348238s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097111179 (3h18m23.900470354s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097545617 (3h18m23.90090479s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097826531 (3h18m23.90118571s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098091522 (3h18m23.901450698s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098582559 (3h18m23.901941739s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098625393 (3h18m23.901984578s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099438197 (3h18m23.902797382s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099473251 (3h18m23.902832435s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.168134648 (3h18m23.971493869s) [upload=3409]: Next returned nil segment
Feb 17 23:34:23.168173410 (3h18m23.97153259s) [upload=3409]: Waiting for error group managing segments...
Feb 17 23:34:23.192662134 (3h18m23.996021325s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.199821483 (3h18m24.003180644s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.216892202 (3h18m24.020251384s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.315136137 (3h18m24.118495316s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.369180508 (3h18m24.17253969s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.676435083 (3h18m24.479794257s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.686494858 (3h18m24.48985402s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690268874 (3h18m24.493628038s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690325658 (3h18m24.493684832s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.702724806 (3h18m24.506083983s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.704052843 (3h18m24.507412028s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:24.098092821 (3h18m24.901451996s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:35:23.025077135 (3h19m23.82843643s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: rpc: dial tcp 77.90.31.15:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025124657 (3h19m23.828483841s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025424430 (3h19m23.828783613s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025719000 (3h19m23.829078184s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025933739 (3h19m23.829292922s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026092417 (3h19m23.829451591s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: write tcp 159.223.143.39:50068->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026120182 (3h19m23.829479349s) [seg_pos={0 0}, upload=3409]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:35:23.049067071 (3h19m23.852426247s) [upload=3409]: Done uploading segments...
Feb 17 23:35:23.049115322 (3h19m23.852474496s) [upload=3409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 938.955183ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..............XX............................................] duration:
517.25937ms trace:
https://jaeger.dev.storj.io/trace/732e062283542b6f
FAIL (2026-02-17 23:39:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 517.282941ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
20 out of 60 is failed (threshold=3) [..................X..XX.XX.X...XX.X..XXXXX.XX.X.X..XX.......] duration:
922.953441ms trace:
https://jaeger.dev.storj.io/trace/2be28be9b880734a
FAIL (2026-02-17 23:39:38 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Feb 17 23:34:23.059166009 (3h18m23.862525184s) [upload=3409]: Starting upload
Feb 17 23:34:23.059410193 (3h18m23.862769371s) [upload=3409]: Done starting upload
Feb 17 23:34:23.059535700 (3h18m23.86289487s) [upload=3409]: Uploading segments...
Feb 17 23:34:23.060262272 (3h18m23.863621456s) [upload=3409]: Got next segment. Inline: false
Feb 17 23:34:23.060415851 (3h18m23.86377504s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:34:23.060424507 (3h18m23.863783691s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:34:23.091662596 (3h18m23.89502178s) [seg_pos={0 0}, upload=3409]: Begin upload segment...
Feb 17 23:34:23.091736818 (3h18m23.895096053s) [seg_pos={0 0}, upload=3409]: Done begin upload segment.
Feb 17 23:34:23.091776467 (3h18m23.895135652s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.091923016 (3h18m23.8952822s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.092846932 (3h18m23.896206109s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093631811 (3h18m23.896990994s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093929468 (3h18m23.897288656s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.094936563 (3h18m23.898295747s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.095529634 (3h18m23.898888818s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096283238 (3h18m23.899642414s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096313780 (3h18m23.899672964s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096989058 (3h18m23.900348238s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097111179 (3h18m23.900470354s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097545617 (3h18m23.90090479s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097826531 (3h18m23.90118571s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098091522 (3h18m23.901450698s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098582559 (3h18m23.901941739s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098625393 (3h18m23.901984578s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099438197 (3h18m23.902797382s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099473251 (3h18m23.902832435s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.168134648 (3h18m23.971493869s) [upload=3409]: Next returned nil segment
Feb 17 23:34:23.168173410 (3h18m23.97153259s) [upload=3409]: Waiting for error group managing segments...
Feb 17 23:34:23.192662134 (3h18m23.996021325s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.199821483 (3h18m24.003180644s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.216892202 (3h18m24.020251384s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.315136137 (3h18m24.118495316s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.369180508 (3h18m24.17253969s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.676435083 (3h18m24.479794257s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.686494858 (3h18m24.48985402s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690268874 (3h18m24.493628038s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690325658 (3h18m24.493684832s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.702724806 (3h18m24.506083983s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.704052843 (3h18m24.507412028s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:24.098092821 (3h18m24.901451996s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:35:23.025077135 (3h19m23.82843643s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: rpc: dial tcp 77.90.31.15:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025124657 (3h19m23.828483841s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025424430 (3h19m23.828783613s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025719000 (3h19m23.829078184s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025933739 (3h19m23.829292922s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026092417 (3h19m23.829451591s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: write tcp 159.223.143.39:50068->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026120182 (3h19m23.829479349s) [seg_pos={0 0}, upload=3409]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:35:23.049067071 (3h19m23.852426247s) [upload=3409]: Done uploading segments...
Feb 17 23:35:23.049115322 (3h19m23.852474496s) [upload=3409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 922.983306ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........X.X................................................] duration:
596.922625ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:39:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 596.953313ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [....................X..X.XX.XXXXX.X...XXXXXXXXX..XX.XX......] duration:
1.219488927s trace:
https://jaeger.dev.storj.io/trace/155330f7fc0e4dbc
FAIL (2026-02-17 23:39:27 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7 23:33:35.296364403 (3h17m36.099723572s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:35.296369828 (3h17m36.099729008s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:35.328531581 (3h17m36.131890757s) [seg_pos={0 0}, upload=3400]: Begin upload segment...
Feb 17 23:33:35.328594851 (3h17m36.131954034s) [seg_pos={0 0}, upload=3400]: Done begin upload segment.
Feb 17 23:33:35.328629116 (3h17m36.131988303s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.329707651 (3h17m36.133066835s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330219639 (3h17m36.133578819s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330709168 (3h17m36.134068348s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331183887 (3h17m36.134543064s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331616342 (3h17m36.134975519s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332168121 (3h17m36.135527298s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332636066 (3h17m36.135995242s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333221709 (3h17m36.136580889s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333738059 (3h17m36.137097242s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334246293 (3h17m36.13760547s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334948323 (3h17m36.138307498s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.335613706 (3h17m36.138972889s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337192803 (3h17m36.140551986s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337747107 (3h17m36.141106281s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338369048 (3h17m36.141728224s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338990672 (3h17m36.142349846s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.339653294 (3h17m36.143012469s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.408953805 (3h17m36.212312973s) [upload=3400]: Next returned nil segment
Feb 17 23:33:35.408965890 (3h17m36.212325065s) [upload=3400]: Waiting for error group managing segments...
Feb 17 23:33:35.418290309 (3h17m36.221649484s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.425756540 (3h17m36.229115704s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.427489628 (3h17m36.230848804s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.461340071 (3h17m36.264699245s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.462428268 (3h17m36.265787445s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.488391421 (3h17m36.291750584s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.923338747 (3h17m36.726697923s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.925771195 (3h17m36.72913037s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.926498548 (3h17m36.729857723s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.966840449 (3h17m36.770199625s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.027376222 (3h17m36.830735406s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.338385603 (3h17m37.14174478s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:34:35.263290924 (3h18m36.066650107s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263414921 (3h18m36.066774097s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12iLnZ6bvgFWKtP79m7pa6rmStrpMGJQeUuDV3kTHQdFZQAWS4, address:77.90.31.16:4752): protocol: write tcp 159.223.143.39:56974->77.90.31.16:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263594757 (3h18m36.066953935s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:41266->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263729993 (3h18m36.067089176s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263772396 (3h18m36.067131576s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: write tcp 159.223.143.39:45336->77.90.31.11:4562: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264125641 (3h18m36.067484825s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: rpc: dial tcp 77.90.31.15:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264162026 (3h18m36.0675212s) [seg_pos={0 0}, upload=3400]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:35.290242339 (3h18m36.09360153s) [upload=3400]: Done uploading segments...
Feb 17 23:34:35.290317435 (3h18m36.093676616s) [upload=3400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1.219518606s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........X.X...............................................] duration:
560.182868ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:39:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 560.211391ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
20 out of 60 is failed (threshold=3) [...................X..XX.XX.X...XX.X..XXXXX.XX.X.X..XX......] duration:
893.827935ms trace:
https://jaeger.dev.storj.io/trace/2be28be9b880734a
FAIL (2026-02-17 23:39:00 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Feb 17 23:34:23.059166009 (3h18m23.862525184s) [upload=3409]: Starting upload
Feb 17 23:34:23.059410193 (3h18m23.862769371s) [upload=3409]: Done starting upload
Feb 17 23:34:23.059535700 (3h18m23.86289487s) [upload=3409]: Uploading segments...
Feb 17 23:34:23.060262272 (3h18m23.863621456s) [upload=3409]: Got next segment. Inline: false
Feb 17 23:34:23.060415851 (3h18m23.86377504s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:34:23.060424507 (3h18m23.863783691s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:34:23.091662596 (3h18m23.89502178s) [seg_pos={0 0}, upload=3409]: Begin upload segment...
Feb 17 23:34:23.091736818 (3h18m23.895096053s) [seg_pos={0 0}, upload=3409]: Done begin upload segment.
Feb 17 23:34:23.091776467 (3h18m23.895135652s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.091923016 (3h18m23.8952822s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.092846932 (3h18m23.896206109s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093631811 (3h18m23.896990994s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093929468 (3h18m23.897288656s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.094936563 (3h18m23.898295747s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.095529634 (3h18m23.898888818s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096283238 (3h18m23.899642414s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096313780 (3h18m23.899672964s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096989058 (3h18m23.900348238s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097111179 (3h18m23.900470354s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097545617 (3h18m23.90090479s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097826531 (3h18m23.90118571s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098091522 (3h18m23.901450698s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098582559 (3h18m23.901941739s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098625393 (3h18m23.901984578s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099438197 (3h18m23.902797382s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099473251 (3h18m23.902832435s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.168134648 (3h18m23.971493869s) [upload=3409]: Next returned nil segment
Feb 17 23:34:23.168173410 (3h18m23.97153259s) [upload=3409]: Waiting for error group managing segments...
Feb 17 23:34:23.192662134 (3h18m23.996021325s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.199821483 (3h18m24.003180644s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.216892202 (3h18m24.020251384s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.315136137 (3h18m24.118495316s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.369180508 (3h18m24.17253969s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.676435083 (3h18m24.479794257s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.686494858 (3h18m24.48985402s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690268874 (3h18m24.493628038s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690325658 (3h18m24.493684832s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.702724806 (3h18m24.506083983s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.704052843 (3h18m24.507412028s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:24.098092821 (3h18m24.901451996s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:35:23.025077135 (3h19m23.82843643s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: rpc: dial tcp 77.90.31.15:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025124657 (3h19m23.828483841s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025424430 (3h19m23.828783613s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025719000 (3h19m23.829078184s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025933739 (3h19m23.829292922s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026092417 (3h19m23.829451591s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: write tcp 159.223.143.39:50068->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026120182 (3h19m23.829479349s) [seg_pos={0 0}, upload=3409]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:35:23.049067071 (3h19m23.852426247s) [upload=3409]: Done uploading segments...
Feb 17 23:35:23.049115322 (3h19m23.852474496s) [upload=3409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 893.860737ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...............XX...........................................] duration:
509.52982ms trace:
https://jaeger.dev.storj.io/trace/732e062283542b6f
FAIL (2026-02-17 23:38:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 509.571608ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........X.X..............................................] duration:
573.863422ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:38:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 573.883033ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [.....................X..X.XX.XXXXX.X...XXXXXXXXX..XX.XX.....] duration:
936.768512ms trace:
https://jaeger.dev.storj.io/trace/155330f7fc0e4dbc
FAIL (2026-02-17 23:38:20 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7 23:33:35.296364403 (3h17m36.099723572s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:35.296369828 (3h17m36.099729008s) [upload=3400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:35.328531581 (3h17m36.131890757s) [seg_pos={0 0}, upload=3400]: Begin upload segment...
Feb 17 23:33:35.328594851 (3h17m36.131954034s) [seg_pos={0 0}, upload=3400]: Done begin upload segment.
Feb 17 23:33:35.328629116 (3h17m36.131988303s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.329707651 (3h17m36.133066835s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330219639 (3h17m36.133578819s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.330709168 (3h17m36.134068348s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331183887 (3h17m36.134543064s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.331616342 (3h17m36.134975519s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332168121 (3h17m36.135527298s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.332636066 (3h17m36.135995242s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333221709 (3h17m36.136580889s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.333738059 (3h17m36.137097242s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334246293 (3h17m36.13760547s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.334948323 (3h17m36.138307498s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.335613706 (3h17m36.138972889s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337192803 (3h17m36.140551986s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.337747107 (3h17m36.141106281s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338369048 (3h17m36.141728224s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.338990672 (3h17m36.142349846s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.339653294 (3h17m36.143012469s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Uploading piece...
Feb 17 23:33:35.408953805 (3h17m36.212312973s) [upload=3400]: Next returned nil segment
Feb 17 23:33:35.408965890 (3h17m36.212325065s) [upload=3400]: Waiting for error group managing segments...
Feb 17 23:33:35.418290309 (3h17m36.221649484s) [noise=true , address=145.79.227.16:4930 , piece_id=VROUA3HD62YYWR6PPT3CVYQU6OISG2OWT7S7AON27MH3TIJS3A5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.425756540 (3h17m36.229115704s) [noise=true , address=145.79.227.28:4743 , piece_id=CQ376H6NIMOP6DTNFSFMKYRHSB55KMXEK35NGLARHYY4GNZCXDMQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.427489628 (3h17m36.230848804s) [noise=true , address=45.91.116.5:4330 , piece_id=IZHARPW4NI4URCGWQXH7EURKPO76T27KUFA2LQJ6WS7OAT7IKVEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.461340071 (3h17m36.264699245s) [noise=true , address=145.79.227.31:4811 , piece_id=X4PVTH2ORJF6YEXWANOPHP34WVM5JKOOLFLFYOPLRSLEDSUFEM4Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.462428268 (3h17m36.265787445s) [noise=true , address=145.79.227.33:4902 , piece_id=QT6CULADWTDDXXS5OGKPMSSUVLLYANJZVIREK6OU75LCJVLOWKQA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.488391421 (3h17m36.291750584s) [noise=true , address=145.79.227.44:4212 , piece_id=KDKQES2B4KLBHDMQVFP73WOR6KUUQ44AWKRNSLNZCB5Q5I4K7V5A, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.923338747 (3h17m36.726697923s) [noise=true , address=193.149.164.69:4083 , piece_id=OQXNGHC7QNZCLQO7AIIJ5JPMNK5SM44H7X5OY5CZVZSKU3D7BKEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.925771195 (3h17m36.72913037s) [noise=true , address=193.149.164.66:4965 , piece_id=5RHXNWMI4D2YH7MPGAYS5OBQG3BAFNCZLQA5P45REAWQV52HRRVA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.926498548 (3h17m36.729857723s) [noise=true , address=64.49.13.27:4966 , piece_id=3WR3ZSZXTKNNASTEXD3REQFVR7GSLEW5QFE3LVDBFCLVUIORGFPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:35.966840449 (3h17m36.770199625s) [noise=true , address=193.149.164.70:4110 , piece_id=MPE6SU6CMIQOR5SN4CSGZR732ZBVHWLVM7PH2ASJEEFQWQB5OJOA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.027376222 (3h17m36.830735406s) [noise=true , address=193.149.164.143:4698 , piece_id=26PHUTHTSGPXCR3WYJTT3BBOA6EJDIFENMO5LUIZT3G2PDXDTRDQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:33:36.338385603 (3h17m37.14174478s) [noise=true , address=145.79.227.30:4795 , piece_id=T2SC5BE3TTVT2XQ7XD5VAOULTKPPJQWR33DLSEJG2NRHDNYQ4JPA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: <nil>
Feb 17 23:34:35.263290924 (3h18m36.066650107s) [noise=true , address=77.90.31.13:4649 , piece_id=IU3J3TUHCYQPNTIWT2K6ACD37B2CPYXNOENLU6HL4GRNG5XBP4UA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, address:77.90.31.13:4649): protocol: rpc: dial tcp 77.90.31.13:4649: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263414921 (3h18m36.066774097s) [noise=true , address=77.90.31.16:4752 , piece_id=JFGKN4TRFE7XBYFKMT5RHXGQ66OXGQEC4PDZHLPUPAYZYOTPPU5Q, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12iLnZ6bvgFWKtP79m7pa6rmStrpMGJQeUuDV3kTHQdFZQAWS4, address:77.90.31.16:4752): protocol: write tcp 159.223.143.39:56974->77.90.31.16:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263594757 (3h18m36.066953935s) [noise=true , address=77.90.31.18:4821 , piece_id=JVYRU7XDYPSXDTNIU7CP3JVVGYUKMJRH6KADPGN6SXUUDRQKDMUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:41266->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263729993 (3h18m36.067089176s) [noise=true , address=77.90.31.21:4887 , piece_id=KHGKL3IC2BJQ6EKR3YNV34ZHWT5PX7UVWRTEZVAOLCA25H7Q6JZA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: rpc: dial tcp 77.90.31.21:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.263772396 (3h18m36.067131576s) [noise=true , address=77.90.31.11:4562 , piece_id=QZYES2HZLU2DUU4QVOLIL2EGFP6VTZ43H7UKDZ3MXGN3UHDSZIEA, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: write tcp 159.223.143.39:45336->77.90.31.11:4562: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264125641 (3h18m36.067484825s) [noise=true , address=77.90.31.15:4701 , piece_id=22PMHYGE66GM5P2OVVSFCZYK4WXKV7HRYDPVBX4GF2BZ4SFSDYUQ, seg_pos={0 0}, upload=3400]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: rpc: dial tcp 77.90.31.15:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:35.264162026 (3h18m36.0675212s) [seg_pos={0 0}, upload=3400]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:35.290242339 (3h18m36.09360153s) [upload=3400]: Done uploading segments...
Feb 17 23:34:35.290317435 (3h18m36.093676616s) [upload=3400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 936.790702ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
20 out of 60 is failed (threshold=3) [....................X..XX.XX.X...XX.X..XXXXX.XX.X.X..XX.....] duration:
546.789935ms trace:
https://jaeger.dev.storj.io/trace/2be28be9b880734a
FAIL (2026-02-17 23:37:59 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Feb 17 23:34:23.059166009 (3h18m23.862525184s) [upload=3409]: Starting upload
Feb 17 23:34:23.059410193 (3h18m23.862769371s) [upload=3409]: Done starting upload
Feb 17 23:34:23.059535700 (3h18m23.86289487s) [upload=3409]: Uploading segments...
Feb 17 23:34:23.060262272 (3h18m23.863621456s) [upload=3409]: Got next segment. Inline: false
Feb 17 23:34:23.060415851 (3h18m23.86377504s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:34:23.060424507 (3h18m23.863783691s) [upload=3409]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:34:23.091662596 (3h18m23.89502178s) [seg_pos={0 0}, upload=3409]: Begin upload segment...
Feb 17 23:34:23.091736818 (3h18m23.895096053s) [seg_pos={0 0}, upload=3409]: Done begin upload segment.
Feb 17 23:34:23.091776467 (3h18m23.895135652s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.091923016 (3h18m23.8952822s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.092846932 (3h18m23.896206109s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093631811 (3h18m23.896990994s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.093929468 (3h18m23.897288656s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.094936563 (3h18m23.898295747s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.095529634 (3h18m23.898888818s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096283238 (3h18m23.899642414s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096313780 (3h18m23.899672964s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.096989058 (3h18m23.900348238s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097111179 (3h18m23.900470354s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097545617 (3h18m23.90090479s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.097826531 (3h18m23.90118571s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098091522 (3h18m23.901450698s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098582559 (3h18m23.901941739s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.098625393 (3h18m23.901984578s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099438197 (3h18m23.902797382s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.099473251 (3h18m23.902832435s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Uploading piece...
Feb 17 23:34:23.168134648 (3h18m23.971493869s) [upload=3409]: Next returned nil segment
Feb 17 23:34:23.168173410 (3h18m23.97153259s) [upload=3409]: Waiting for error group managing segments...
Feb 17 23:34:23.192662134 (3h18m23.996021325s) [noise=true , address=45.91.116.5:4309 , piece_id=333IVACIE6CXBMQJHIP4RFOVY7LGHZTO6ND6VZMKRJ5OCC47HJ2A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.199821483 (3h18m24.003180644s) [noise=true , address=145.79.227.36:4545 , piece_id=WCTB77G5E3O4BJXBIIZBX2ND24JDT7C2TRQN3GHWOTTI6NW26YVA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.216892202 (3h18m24.020251384s) [noise=true , address=45.91.116.8:4069 , piece_id=LAKFDRQDCLMCFMCDEKKQNRUAPTBABAXCDI4C3EQN3SNJWLT6VDOA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.315136137 (3h18m24.118495316s) [noise=true , address=5.231.73.4:4014 , piece_id=6MST2M2SIGDVNRG5GEVCIFXIUIZZAGECO6PXJA3JKIMWROJJZYMQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.369180508 (3h18m24.17253969s) [noise=true , address=145.79.227.20:4470 , piece_id=R2QNZD4ZJ4DWIBYWN5WGEGFQJVHYSEKOKDP3JW3CHIVE3GMVKZHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.676435083 (3h18m24.479794257s) [noise=true , address=64.49.13.27:4958 , piece_id=WNCSGKSGEXYSQVN75WUVLEJGRXTF4SFYHYIOSHBLW3PNDLKT5M4Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.686494858 (3h18m24.48985402s) [noise=true , address=193.149.164.66:4973 , piece_id=ZVLUSZ367TBLNKNXTDFX2HBSQWFGG3DEJOQL23MQUCIPTP5Y5DIQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690268874 (3h18m24.493628038s) [noise=true , address=193.149.164.68:4068 , piece_id=NT5UQR6I755XZ6O2IKSG6PFFG5CSNCX67726EHHB6PQAB5PGX24A, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.690325658 (3h18m24.493684832s) [noise=true , address=193.149.164.70:4118 , piece_id=5NNF2HK3VETFSS6X7LK7E5FBQBWTK2XRFSWGEUE72KHRBNLEG7LQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.702724806 (3h18m24.506083983s) [noise=true , address=193.149.164.69:4099 , piece_id=J2MFQU3LFBWBLMNU44PWHAAJYANHXKXUANGW4IFXBW2NEPSRVAYA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:23.704052843 (3h18m24.507412028s) [noise=true , address=193.149.164.67:4998 , piece_id=776VEOKL25N2MNQWORHJVZIMY6CKZCQD26OVG5YB4MOZMJHGA5IA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:34:24.098092821 (3h18m24.901451996s) [noise=true , address=145.79.227.30:4792 , piece_id=A56BKPGS6324OW26YUAOL2BWWFZMFTQYUONMD6RXFHBNHQI7LQNA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: <nil>
Feb 17 23:35:23.025077135 (3h19m23.82843643s) [noise=true , address=77.90.31.15:4732 , piece_id=SAFTUJRI4M3KMND6MGBTHNHNRFJRXLJZI5XLVNF73GKIQAKW4IUQ, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: rpc: dial tcp 77.90.31.15:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025124657 (3h19m23.828483841s) [noise=true , address=77.90.31.17:4778 , piece_id=J7IMFHFZILRC5MAOSIPKU6R4HHNMUUVDY6PURCQDV3YYHMZCPMPA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025424430 (3h19m23.828783613s) [noise=true , address=77.90.31.13:4658 , piece_id=4XHR6Y2AGXAU5NVFWMX6VVLVBEMC34O2RBCLCHOYEFCOFTINDAKA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: rpc: dial tcp 77.90.31.13:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025719000 (3h19m23.829078184s) [noise=true , address=77.90.31.19:4871 , piece_id=DTW2SQEFMMJD76PX4IDYFOSDMC7HYDCQ3WLCJYNZFC2ITQTNC77Q, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.025933739 (3h19m23.829292922s) [noise=true , address=77.90.31.18:4823 , piece_id=5EMG3QQSBTJY6ZIE6GR6HRRVCY763Y63TSRZ3A5WYQ2HZIISGBHA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026092417 (3h19m23.829451591s) [noise=true , address=77.90.31.21:4899 , piece_id=XIYMFGCBESO7KAASS76HSLC7EPJFMM6AXAPICV77PLTWV5LBWWQA, seg_pos={0 0}, upload=3409]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: write tcp 159.223.143.39:50068->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:35:23.026120182 (3h19m23.829479349s) [seg_pos={0 0}, upload=3409]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:35:23.049067071 (3h19m23.852426247s) [upload=3409]: Done uploading segments...
Feb 17 23:35:23.049115322 (3h19m23.852474496s) [upload=3409]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 546.821341ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [................XX..........................................] duration:
481.037698ms trace:
https://jaeger.dev.storj.io/trace/732e062283542b6f
FAIL (2026-02-17 23:37:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 481.065146ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............X.X.............................................] duration:
576.210857ms trace:
https://jaeger.dev.storj.io/trace/22cf127fc9cd3d45
FAIL (2026-02-17 23:37:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 576.236461ms
core
uplink
us-select-1
us1-placement-20
|